2024-08-30 19:59:24,190 INFO StreamThr :29126 [internal.py:wandb_internal():85] W&B internal server running at pid: 29126, started at: 2024-08-30 19:59:24.188690 2024-08-30 19:59:24,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status 2024-08-30 19:59:24,197 INFO WriterThread:29126 [datastore.py:open_for_write():87] open: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/run-mao0tqjy.wandb 2024-08-30 19:59:24,198 DEBUG SenderThread:29126 [sender.py:send():391] send: header 2024-08-30 19:59:24,201 DEBUG SenderThread:29126 [sender.py:send():391] send: run 2024-08-30 19:59:24,579 INFO SenderThread:29126 [dir_watcher.py:__init__():211] watching files in: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files 2024-08-30 19:59:24,579 INFO SenderThread:29126 [sender.py:_start_run_threads():1200] run started: mao0tqjy with start time 1725047964.187209 2024-08-30 19:59:24,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: run_start 2024-08-30 19:59:24,607 DEBUG HandlerThread:29126 [system_info.py:__init__():26] System info init 2024-08-30 19:59:24,607 DEBUG HandlerThread:29126 [system_info.py:__init__():41] System info init done 2024-08-30 19:59:24,607 INFO HandlerThread:29126 [system_monitor.py:start():194] Starting system monitor 2024-08-30 19:59:24,608 INFO SystemMonitor:29126 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-08-30 19:59:24,608 INFO HandlerThread:29126 [system_monitor.py:probe():214] Collecting system info 2024-08-30 19:59:24,610 INFO SystemMonitor:29126 [interfaces.py:start():188] Started cpu monitoring 2024-08-30 19:59:24,612 INFO SystemMonitor:29126 [interfaces.py:start():188] Started disk monitoring 2024-08-30 19:59:24,614 INFO SystemMonitor:29126 [interfaces.py:start():188] Started gpu monitoring 2024-08-30 19:59:24,615 INFO SystemMonitor:29126 [interfaces.py:start():188] Started memory monitoring 2024-08-30 19:59:24,616 INFO SystemMonitor:29126 [interfaces.py:start():188] Started network monitoring 2024-08-30 19:59:24,641 DEBUG HandlerThread:29126 [system_info.py:probe():152] Probing system 2024-08-30 19:59:24,643 DEBUG HandlerThread:29126 [system_info.py:_probe_git():137] Probing git 2024-08-30 19:59:24,654 DEBUG HandlerThread:29126 [system_info.py:_probe_git():145] Probing git done 2024-08-30 19:59:24,654 DEBUG HandlerThread:29126 [system_info.py:probe():200] Probing system done 2024-08-30 19:59:24,654 DEBUG HandlerThread:29126 [system_monitor.py:probe():223] {'os': 'Linux-6.5.0-41-generic-x86_64-with-glibc2.35', 'python': '3.11.9', 'heartbeatAt': '2024-08-30T19:59:24.641329', 'startedAt': '2024-08-30T19:59:24.177472', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '-m nanoT5.main', 'codePathLocal': None, 'git': {'remote': 'https://github.com/pszemraj/nanoT5.git', 'commit': '58834d398cca39b8344c83490f8b1bec71116423'}, 'email': None, 'root': '/workspace/nanoT5', 'host': 'cf696b887dc2', 'username': 'root', 'executable': '/usr/bin/python', 'cpu_count': 100, 'cpu_count_logical': 100, 'cpu_freq': {'current': 2249.869999999997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}, {'current': 2249.87, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 200.0, 'used': 1.7721595764160156}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 668.8548545837402}} 2024-08-30 19:59:24,654 INFO HandlerThread:29126 [system_monitor.py:probe():224] Finished collecting system info 2024-08-30 19:59:24,654 INFO HandlerThread:29126 [system_monitor.py:probe():227] Publishing system info 2024-08-30 19:59:24,655 DEBUG HandlerThread:29126 [system_info.py:_save_code():44] Saving code 2024-08-30 19:59:24,655 WARNING HandlerThread:29126 [system_info.py:_save_code():46] unable to save code -- program entry not found 2024-08-30 19:59:24,655 DEBUG HandlerThread:29126 [system_info.py:_save_patches():84] Saving git patches 2024-08-30 19:59:24,684 DEBUG HandlerThread:29126 [system_info.py:_save_patches():126] Saving git patches done 2024-08-30 19:59:24,685 INFO HandlerThread:29126 [system_monitor.py:probe():229] Finished publishing system info 2024-08-30 19:59:24,690 DEBUG SenderThread:29126 [sender.py:send():391] send: files 2024-08-30 19:59:24,690 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-metadata.json with policy now 2024-08-30 19:59:24,690 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file diff.patch with policy now 2024-08-30 19:59:24,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: python_packages 2024-08-30 19:59:24,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 19:59:24,814 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: python_packages 2024-08-30 19:59:24,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 19:59:24,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 19:59:24,961 DEBUG SenderThread:29126 [sender.py:send():391] send: telemetry 2024-08-30 19:59:25,170 INFO wandb-upload_1:29126 [upload_job.py:push():130] Uploaded file /tmp/tmp4lmrcxuywandb/gu36bmbx-diff.patch 2024-08-30 19:59:25,205 INFO wandb-upload_0:29126 [upload_job.py:push():130] Uploaded file /tmp/tmp4lmrcxuywandb/recxkp27-wandb-metadata.json 2024-08-30 19:59:25,580 INFO Thread-12 :29126 [dir_watcher.py:_on_file_created():271] file/dir created: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-metadata.json 2024-08-30 19:59:25,580 INFO Thread-12 :29126 [dir_watcher.py:_on_file_created():271] file/dir created: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:25,581 INFO Thread-12 :29126 [dir_watcher.py:_on_file_created():271] file/dir created: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/requirements.txt 2024-08-30 19:59:25,581 INFO Thread-12 :29126 [dir_watcher.py:_on_file_created():271] file/dir created: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/diff.patch 2024-08-30 19:59:27,580 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:29,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:29,581 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:31,582 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:33,582 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:34,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 19:59:34,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:39,585 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:39,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 19:59:39,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 19:59:40,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:43,586 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:44,798 DEBUG SenderThread:29126 [sender.py:send():391] send: config 2024-08-30 19:59:44,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 19:59:45,587 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:45,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:47,588 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 19:59:50,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:54,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 19:59:54,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 19:59:54,996 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 19:59:56,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 19:59:57,597 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/config.yaml 2024-08-30 20:00:01,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:04,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:06,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:09,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:00:09,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:00:12,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:14,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:17,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:22,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:24,691 DEBUG SystemMonitor:29126 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-08-30 20:00:24,696 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:00:24,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:24,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:00:24,996 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:00:28,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:34,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:34,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:39,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:39,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:00:39,997 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:00:44,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:44,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:50,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:00:54,699 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:00:54,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:00:54,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:00:54,997 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:00:56,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:01,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:04,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:06,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:10,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:01:10,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:01:11,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:14,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:17,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:22,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:24,701 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:01:24,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:25,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:01:25,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:01:27,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:32,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:34,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:37,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:40,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:01:40,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:01:42,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:44,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:47,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:53,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:01:54,703 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:01:54,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:01:55,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:01:55,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:01:58,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:03,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:04,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:08,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:10,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:02:10,020 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:02:13,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:14,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:18,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:24,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:24,709 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:02:24,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:25,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:02:25,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:02:29,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:34,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:34,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:39,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:40,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:02:40,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:02:44,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:45,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:51,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:02:54,709 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:02:54,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:02:55,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:02:55,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:02:56,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:01,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:04,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:06,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:10,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:03:10,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:03:12,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:14,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:17,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:23,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:24,712 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:03:24,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:25,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:03:25,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:03:28,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:34,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:34,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:36,084 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:03:39,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:40,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:03:40,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:03:44,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:45,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:46,090 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:03:50,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:54,715 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:03:54,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:03:55,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:03:55,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:03:56,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:03:58,097 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:04:01,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:04,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:06,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:10,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:04:10,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:04:12,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:14,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:18,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:24,717 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:04:24,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:24,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:25,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:04:25,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:04:29,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:34,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:35,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:40,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:04:40,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:04:41,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:44,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:46,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:51,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:04:54,722 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:04:54,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:04:55,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:04:55,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:04:56,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:02,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:04,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:08,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:10,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:05:10,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:05:13,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:14,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:19,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:24,723 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:05:24,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:25,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:05:25,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:05:25,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:30,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:34,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:35,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:40,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:05:40,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:05:41,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:44,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:46,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:52,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:05:54,726 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:05:54,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:05:55,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:05:55,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:05:58,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:03,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:04,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:09,360 DEBUG SenderThread:29126 [sender.py:send():391] send: telemetry 2024-08-30 20:06:09,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:09,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:06:10,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:06:10,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:06:11,165 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:06:14,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:14,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:20,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:24,728 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:06:24,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:25,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:06:25,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:06:26,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:27,173 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/config.yaml 2024-08-30 20:06:31,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:34,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:36,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:40,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:06:40,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:06:42,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:44,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:48,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:53,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:06:54,731 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:06:54,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:06:55,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:06:55,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:06:59,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:04,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:05,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:10,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:07:10,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:07:11,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:14,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:16,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:22,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:24,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:07:24,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:25,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:07:25,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:07:28,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:33,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:34,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:38,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:40,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:07:40,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:07:43,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:44,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:49,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:54,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:07:54,736 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:07:54,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:07:55,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:07:55,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:08:00,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:04,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:05,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:10,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:08:10,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:08:11,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:14,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:08:14,534 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:08:14,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:08:14,537 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:08:14,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:15,234 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:08:15,234 INFO Thread-12 :29126 [dir_watcher.py:_on_file_created():271] file/dir created: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:08:16,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:22,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:24,738 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:08:24,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:25,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:08:25,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:08:28,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:33,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:34,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:39,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:40,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:08:40,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:08:44,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:44,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:50,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:08:54,742 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:08:54,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:08:55,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:08:55,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:08:56,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:01,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:04,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:07,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:10,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:09:10,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:09:13,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:14,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:18,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:24,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:24,743 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:09:24,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:25,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:09:25,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:09:29,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:35,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:35,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:40,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:09:40,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:09:41,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:45,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:46,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:51,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:09:54,745 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:09:55,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:09:55,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:09:55,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:09:57,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:02,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:05,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:08,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:10,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:10:10,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:10:13,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:15,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:16,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:10:16,615 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:10:16,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:10:16,616 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:10:17,306 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:10:17,307 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:10:19,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:24,748 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:10:24,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:25,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:25,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:10:25,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:10:30,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:35,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:35,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:40,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:10:40,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:10:41,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:45,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:46,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:51,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:10:54,749 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:10:55,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:10:55,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:10:55,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:10:57,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:02,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:05,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:07,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:10,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:11:10,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:11:13,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:15,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:18,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:24,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:24,752 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:11:25,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:25,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:11:25,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:11:29,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:34,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:35,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:40,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:11:40,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:11:40,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:45,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:45,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:50,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:11:54,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:11:55,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:11:55,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:11:55,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:11:56,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:01,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:05,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:07,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:10,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:12:10,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:12:13,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:15,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:18,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:12:18,423 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:12:18,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:12:18,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:18,424 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:12:19,358 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:12:19,358 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:12:23,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:24,757 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:12:25,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:25,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:12:25,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:12:29,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:34,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:35,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:39,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:40,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:12:40,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:12:45,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:45,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:51,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:12:54,762 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:12:55,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:12:55,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:12:55,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:12:57,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:02,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:05,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:08,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:10,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:13:10,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:13:14,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:15,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:19,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:24,763 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:13:24,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:25,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:25,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:13:25,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:13:30,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:35,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:36,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:40,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:13:40,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:13:42,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:45,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:47,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:53,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:13:54,765 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:13:55,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:13:55,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:13:55,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:13:59,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:04,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:05,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:10,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:14:10,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:14:10,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:15,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:15,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:20,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:21,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:14:21,231 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:14:21,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:14:21,233 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:14:21,410 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:14:23,411 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:14:24,768 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:14:25,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:14:25,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:14:25,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:26,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:31,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:35,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:37,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:40,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:14:40,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:14:43,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:45,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:48,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:53,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:14:54,770 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:14:55,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:14:55,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:14:55,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:14:59,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:04,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:05,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:15:09,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:10,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:15:10,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:15:15,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:15,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:15:21,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:24,773 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:15:25,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:15:25,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:15:25,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:15:27,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:32,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:35,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:15:38,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:40,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:15:40,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:15:44,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:45,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:15:49,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:54,775 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:15:54,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:15:55,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:15:55,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:15:55,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:00,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:05,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:06,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:10,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:16:10,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:16:12,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:15,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:17,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:22,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:16:22,810 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:16:22,811 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:16:22,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:22,812 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:16:23,470 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:16:23,470 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:16:24,783 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:16:25,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:16:25,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:16:25,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:28,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:33,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:35,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:38,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:40,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:16:40,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:16:43,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:45,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:48,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:54,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:16:54,780 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:16:55,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:16:55,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:16:55,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:16:59,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:05,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:05,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:10,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:17:10,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:17:11,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:15,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:16,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:21,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:24,783 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:17:25,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:17:25,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:17:25,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:27,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:32,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:35,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:37,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:40,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:17:40,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:17:43,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:45,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:48,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:53,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:17:54,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:17:55,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:17:55,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:17:55,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:17:58,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:04,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:05,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:10,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:18:10,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:18:10,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:15,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:15,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:21,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:24,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:18:24,501 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:18:24,502 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:18:24,503 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:18:24,531 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:18:24,786 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:18:25,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:18:25,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:18:25,533 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:18:25,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:27,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:32,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:35,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:38,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:40,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:18:40,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:18:44,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:45,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:50,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:18:54,788 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:18:55,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:18:55,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:18:55,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:18:56,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:01,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:05,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:06,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:10,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:19:10,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:19:12,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:15,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:17,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:22,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:24,791 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:19:25,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:19:25,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:19:25,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:28,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:33,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:35,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:38,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:40,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:19:40,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:19:43,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:45,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:48,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:54,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:19:54,793 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:19:55,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:19:55,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:19:55,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:19:59,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:05,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:05,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:10,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:20:10,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:20:11,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:15,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:16,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:21,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:24,795 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:20:25,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:20:25,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:20:25,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:27,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:27,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:20:27,641 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:20:27,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:20:27,643 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:20:28,595 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:20:29,597 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:20:32,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:35,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:38,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:40,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:20:40,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:20:43,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:45,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:48,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:54,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:20:54,802 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:20:55,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:20:55,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:20:55,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:20:59,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:04,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:05,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:21:10,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:21:10,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:21:10,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:15,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:15,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:21:21,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:24,864 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:21:25,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:21:25,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:21:25,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:21:27,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:32,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:35,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:21:37,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:40,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:21:40,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:21:43,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:45,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:21:48,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:54,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:21:54,867 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:21:55,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:21:55,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:21:55,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:00,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:05,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:05,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:10,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:22:10,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:22:11,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:15,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:16,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:21,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:24,870 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:22:25,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:22:25,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:22:25,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:27,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:29,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:22:29,320 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:22:29,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:22:29,321 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:22:29,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:22:31,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:22:32,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:35,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:37,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:40,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:22:40,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:22:43,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:45,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:48,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:54,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:22:54,871 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:22:55,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:22:55,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:22:55,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:22:59,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:05,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:05,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:23:10,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:23:10,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:23:11,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:15,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:23:16,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:21,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:24,874 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:23:25,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:23:25,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:23:25,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:23:27,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:32,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:35,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:23:38,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:40,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:23:40,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:23:43,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:45,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:23:49,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:54,876 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:23:54,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:23:55,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:23:55,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:23:55,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:00,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:05,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:06,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:10,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:24:10,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:24:12,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:15,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:17,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:23,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:24,879 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:24:25,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:24:25,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:24:25,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:28,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:30,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:24:30,867 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:24:30,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:24:30,868 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:24:31,712 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:24:31,713 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:24:34,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:35,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:40,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:24:40,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:24:40,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:45,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:45,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:51,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:24:54,881 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:24:55,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:24:55,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:24:55,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:24:57,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:02,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:05,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:07,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:10,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:25:10,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:25:13,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:15,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:18,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:24,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:24,883 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:25:25,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:25:25,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:25:25,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:30,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:35,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:35,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:40,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:25:40,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:25:41,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:45,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:46,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:52,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:25:54,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:25:55,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:25:55,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:25:55,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:25:58,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:03,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:05,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:08,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:10,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:26:10,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:26:14,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:15,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:19,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:24,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:24,888 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:26:25,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:26:25,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:26:25,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:29,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:33,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:26:33,837 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:26:33,838 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:26:33,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:26:34,771 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:26:34,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:35,772 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:26:35,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:40,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:26:40,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:26:40,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:45,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:45,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:50,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:26:54,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:26:55,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:26:55,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:26:55,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:26:56,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:01,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:05,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:07,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:10,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:27:10,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:27:13,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:15,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:18,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:24,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:24,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:27:25,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:27:25,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:27:25,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:29,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:35,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:35,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:40,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:27:40,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:27:41,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:45,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:46,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:51,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:27:54,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:27:55,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:27:55,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:27:55,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:27:57,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:02,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:05,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:08,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:10,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:28:10,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:28:14,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:15,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:19,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:24,897 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:28:24,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:25,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:28:25,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:28:25,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:30,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:35,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:28:35,217 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:28:35,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:28:35,218 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:28:35,831 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:28:35,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:36,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:37,833 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:28:40,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:28:40,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:28:41,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:45,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:46,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:51,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:28:54,901 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:28:55,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:28:55,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:28:55,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:28:57,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:02,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:05,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:08,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:10,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:29:10,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:29:13,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:15,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:18,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:24,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:24,903 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:29:25,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:29:25,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:29:25,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:29,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:34,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:35,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:39,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:40,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:29:40,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:29:44,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:45,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:50,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:29:54,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:29:55,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:29:55,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:29:55,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:29:56,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:01,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:05,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:06,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:10,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:30:10,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:30:12,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:15,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:17,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:23,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:24,911 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:30:25,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:30:25,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:30:25,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:29,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:34,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:35,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:36,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:30:36,884 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:30:36,884 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:30:36,885 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:30:36,894 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:30:37,896 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:30:40,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:30:40,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:30:40,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:45,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:45,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:51,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:30:54,910 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:30:55,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:30:55,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:30:55,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:30:57,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:02,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:05,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:07,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:10,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:31:10,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:31:13,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:15,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:18,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:23,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:24,915 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:31:25,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:31:25,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:31:25,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:28,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:34,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:35,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:40,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:31:40,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:31:40,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:45,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:45,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:50,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:31:54,915 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:31:55,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:31:55,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:31:55,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:31:56,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:01,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:05,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:06,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:10,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:32:10,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:32:12,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:15,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:17,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:22,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:24,918 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:32:25,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:32:25,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:32:25,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:28,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:33,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:35,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:38,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:32:38,374 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:32:38,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:32:38,375 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:32:38,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:38,950 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:32:39,952 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:32:40,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:32:40,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:32:43,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:45,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:48,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:54,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:32:54,921 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:32:55,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:32:55,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:32:55,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:32:59,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:05,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:05,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:10,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:33:10,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:33:11,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:15,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:16,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:22,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:24,923 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:33:25,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:33:25,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:33:25,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:28,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:33,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:35,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:39,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:40,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:33:40,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:33:45,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:45,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:50,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:33:54,926 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:33:55,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:33:55,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:33:55,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:33:56,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:01,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:05,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:07,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:10,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:34:10,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:34:13,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:15,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:18,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:23,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:24,928 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:34:25,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:34:25,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:34:25,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:28,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:33,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:35,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:38,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:40,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:34:40,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:34:41,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:34:41,601 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:34:41,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:34:41,602 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:34:41,999 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:34:43,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:44,001 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:34:45,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:48,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:53,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:34:54,935 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:34:55,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:34:55,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:34:55,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:34:59,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:05,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:05,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:10,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:35:10,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:35:11,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:15,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:16,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:22,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:24,933 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:35:25,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:35:25,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:35:25,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:28,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:33,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:35,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:38,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:40,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:35:40,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:35:44,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:45,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:50,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:35:54,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:35:55,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:35:55,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:35:55,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:35:56,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:01,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:05,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:07,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:10,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:36:10,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:36:13,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:15,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:18,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:23,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:24,938 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:36:25,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:36:25,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:36:25,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:28,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:34,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:35,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:39,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:40,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:36:40,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:36:42,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:36:42,928 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:36:42,929 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:36:42,930 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:36:43,046 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:36:44,047 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:36:44,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:45,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:50,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:36:54,940 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:36:55,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:36:55,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:36:55,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:36:56,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:01,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:05,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:06,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:10,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:37:10,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:37:12,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:15,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:17,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:22,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:24,943 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:37:25,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:37:25,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:37:25,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:28,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:33,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:35,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:38,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:40,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:37:40,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:37:43,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:45,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:48,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:54,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:37:54,945 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:37:55,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:37:55,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:37:55,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:37:59,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:05,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:05,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:10,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:38:10,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:38:14,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:15,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:20,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:24,947 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:38:25,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:38:25,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:38:25,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:26,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:31,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:35,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:36,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:40,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:38:40,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:38:42,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:44,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:38:44,347 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:38:44,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:38:44,348 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:38:45,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:38:45,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:46,102 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:38:47,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:52,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:38:54,950 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:38:55,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:38:55,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:38:55,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:38:58,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:03,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:05,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:08,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:10,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:39:10,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:39:13,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:15,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:18,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:24,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:24,952 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:39:25,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:39:25,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:39:25,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:29,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:34,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:35,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:40,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:39:40,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:39:40,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:45,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:45,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:51,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:39:54,958 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:39:55,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:39:55,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:39:55,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:39:57,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:02,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:05,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:07,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:10,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:40:10,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:40:13,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:15,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:18,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:24,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:24,956 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:40:25,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:40:25,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:40:25,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:29,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:34,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:35,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:40,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:40:40,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:40:40,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:45,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:45,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:47,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:40:47,236 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:40:47,236 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:40:47,239 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:40:48,164 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:40:48,164 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:40:51,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:40:54,959 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:40:55,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:40:55,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:40:55,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:40:57,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:02,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:06,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:07,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:10,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:41:10,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:41:13,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:16,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:18,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:23,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:24,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:41:25,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:41:25,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:41:26,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:29,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:34,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:36,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:40,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:41:40,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:41:40,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:45,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:46,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:50,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:41:54,962 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:41:55,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:41:55,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:41:56,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:41:56,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:01,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:06,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:06,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:10,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:42:10,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:42:12,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:16,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:17,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:22,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:24,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:42:25,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:42:25,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:42:26,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:28,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:33,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:36,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:39,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:40,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:42:40,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:42:45,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:46,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:48,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:42:48,665 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:42:48,666 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:42:48,667 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:42:49,215 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:42:50,216 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:42:51,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:42:54,967 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:42:55,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:42:55,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:42:56,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:42:57,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:02,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:06,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:07,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:10,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:43:10,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:43:13,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:16,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:18,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:23,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:24,970 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:43:25,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:43:25,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:43:26,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:28,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:34,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:36,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:40,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:43:40,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:43:40,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:45,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:46,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:50,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:43:54,971 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:43:55,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:43:55,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:43:56,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:43:56,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:01,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:06,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:07,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:10,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:44:10,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:44:13,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:16,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:18,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:23,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:24,978 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:44:25,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:44:25,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:44:26,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:28,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:34,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:36,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:39,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:40,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:44:40,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:44:45,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:46,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:50,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:44:50,115 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:44:50,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:44:50,116 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:44:50,272 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:44:51,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:44:52,274 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:44:54,976 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:44:55,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:44:55,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:44:56,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:44:56,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:01,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:06,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:07,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:10,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:45:10,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:45:13,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:16,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:18,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:24,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:24,981 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:45:25,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:45:25,131 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:45:26,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:29,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:34,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:36,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:40,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:45:40,131 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:45:40,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:45,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:46,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:51,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:45:54,981 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:45:55,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:45:55,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:45:56,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:45:57,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:02,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:06,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:07,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:10,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:46:10,131 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:46:13,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:16,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:18,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:23,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:24,984 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:46:25,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:46:25,131 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:46:26,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:29,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:34,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:36,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:40,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:46:40,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:46:40,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:45,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:46,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:50,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:54,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:46:54,283 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:46:54,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:46:54,284 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:46:54,332 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:46:54,986 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:46:55,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:46:55,131 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:46:55,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:46:56,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:46:56,333 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:47:00,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:06,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:06,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:10,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:47:10,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:47:12,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:16,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:17,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:23,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:24,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:47:25,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:47:25,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:47:26,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:29,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:34,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:36,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:40,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:47:40,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:47:40,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:45,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:46,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:51,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:47:54,992 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:47:55,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:47:55,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:47:56,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:47:56,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:01,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:06,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:07,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:10,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:48:10,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:48:13,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:16,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:18,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:23,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:24,994 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:48:25,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:48:25,133 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:48:26,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:29,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:35,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:36,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:40,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:48:40,133 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:48:40,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:45,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:46,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:51,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:55,000 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:48:55,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:48:55,135 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:48:55,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:48:55,733 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:48:55,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:48:55,734 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:48:56,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:48:56,396 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:48:56,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:48:58,398 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:49:02,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:06,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:49:07,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:10,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:49:10,135 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:49:12,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:16,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:49:18,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:24,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:24,999 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:49:25,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:49:25,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:49:26,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:49:29,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:34,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:36,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:49:39,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:40,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:49:40,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:49:44,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:46,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:49:49,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:54,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:49:55,003 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:49:55,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:49:55,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:49:56,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:00,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:05,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:06,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:10,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:50:10,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:50:11,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:16,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:16,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:22,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:25,004 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:50:25,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:50:25,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:50:26,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:28,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:34,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:36,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:40,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:50:40,137 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:50:40,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:45,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:46,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:51,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:55,007 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:50:55,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:50:55,137 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:50:56,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:50:56,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:50:57,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:50:57,239 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:50:57,239 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:50:57,240 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:50:57,449 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:50:58,449 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:51:02,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:06,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:08,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:10,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:51:10,137 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:51:14,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:16,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:20,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:25,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:51:25,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:51:25,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:51:26,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:26,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:31,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:36,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:36,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:40,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:51:40,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:51:42,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:46,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:48,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:53,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:51:55,013 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:51:55,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:51:55,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:51:56,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:51:59,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:04,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:06,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:09,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:10,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:52:10,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:52:15,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:16,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:21,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:25,015 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:52:25,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:52:25,139 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:52:26,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:26,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:31,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:36,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:37,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:40,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:52:40,139 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:52:43,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:46,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:48,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:53,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:52:55,018 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:52:55,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:52:55,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:52:56,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:52:58,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:00,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:53:00,170 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:53:00,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:53:00,171 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:53:00,515 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:53:02,516 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:53:04,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:06,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:10,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:53:10,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:53:10,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:15,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:16,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:21,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:25,024 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:53:25,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:53:25,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:53:26,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:27,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:32,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:36,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:38,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:40,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:53:40,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:53:44,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:46,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:50,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:53:55,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:53:55,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:53:55,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:53:56,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:53:56,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:01,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:06,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:07,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:10,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:54:10,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:54:12,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:16,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:17,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:23,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:25,026 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:54:25,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:54:25,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:54:26,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:29,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:35,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:36,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:40,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:54:40,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:54:41,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:46,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:47,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:52,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:54:55,026 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:54:55,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:54:55,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:54:56,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:54:58,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:01,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:55:01,603 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:55:01,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:55:01,604 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:55:02,587 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:55:02,587 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:55:03,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:06,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:09,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:10,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:55:10,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:55:14,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:16,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:19,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:24,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:25,028 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:55:25,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:55:25,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:55:26,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:30,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:35,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:36,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:40,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:55:40,142 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:55:41,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:46,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:46,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:52,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:55:55,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:55:55,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:55:55,142 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:55:56,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:55:58,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:03,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:06,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:08,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:10,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:56:10,143 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:56:14,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:16,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:20,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:25,033 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:56:25,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:56:25,143 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:56:26,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:26,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:31,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:36,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:36,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:40,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:56:40,144 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:56:42,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:46,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:47,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:52,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:56:55,035 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:56:55,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:56:55,144 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:56:56,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:56:57,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:02,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:57:02,934 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:57:02,934 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:57:02,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:02,935 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:57:03,657 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:57:04,658 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:57:06,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:08,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:10,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:57:10,145 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:57:14,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:16,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:19,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:25,038 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:57:25,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:25,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:57:25,144 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:57:26,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:30,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:35,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:36,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:40,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:57:40,144 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:57:41,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:46,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:46,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:51,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:57:55,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:57:55,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:57:55,145 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:57:56,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:57:57,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:03,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:06,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:08,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:10,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:58:10,145 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:58:14,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:16,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:19,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:25,047 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:58:25,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:25,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:58:25,146 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:58:26,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:30,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:35,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:36,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:40,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:58:40,145 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:58:41,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:46,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:47,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:52,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:58:55,045 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:58:55,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:58:55,146 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:58:56,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:58:58,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:03,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:05,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 20:59:05,726 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 20:59:05,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 20:59:05,729 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 20:59:05,730 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 20:59:06,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:06,731 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 20:59:08,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:10,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:59:10,146 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:59:14,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:16,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:20,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:25,048 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:59:25,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:59:25,146 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:59:25,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:26,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:30,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:36,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:36,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:40,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:59:40,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:59:41,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:46,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:47,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:52,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 20:59:55,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 20:59:55,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 20:59:55,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 20:59:56,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 20:59:57,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:03,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:06,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:09,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:10,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:00:10,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:00:15,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:16,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:20,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:25,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:00:25,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:00:25,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:00:26,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:26,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:31,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:36,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:36,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:40,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:00:40,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:00:42,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:46,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:48,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:53,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:00:55,055 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:00:55,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:00:55,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:00:56,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:00:58,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:03,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:06,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:01:07,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:01:07,154 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:01:07,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:01:07,155 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:01:07,805 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:01:08,806 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:01:09,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:10,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:01:10,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:01:14,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:16,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:01:20,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:25,057 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:01:25,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:01:25,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:01:26,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:01:26,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:32,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:36,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:01:38,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:40,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:01:40,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:01:44,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:46,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:01:49,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:55,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:01:55,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:01:55,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:01:55,149 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:01:56,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:00,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:06,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:06,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:10,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:02:10,150 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:02:12,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:16,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:18,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:23,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:25,061 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:02:25,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:02:25,149 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:02:26,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:28,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:33,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:36,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:39,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:40,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:02:40,149 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:02:45,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:46,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:51,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:02:55,067 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:02:55,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:02:55,150 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:02:56,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:02:57,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:02,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:06,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:07,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:08,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:03:08,699 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:03:08,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:03:08,700 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:03:08,871 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:03:10,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:03:10,150 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:03:10,872 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:03:13,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:16,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:18,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:24,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:25,068 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:03:25,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:03:25,150 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:03:26,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:30,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:35,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:36,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:40,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:03:40,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:03:41,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:46,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:46,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:51,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:03:55,070 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:03:55,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:03:55,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:03:56,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:03:57,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:02,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:06,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:08,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:10,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:04:10,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:04:14,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:16,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:19,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:25,073 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:04:25,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:25,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:04:25,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:04:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:30,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:36,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:36,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:40,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:04:40,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:04:41,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:46,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:47,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:52,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:04:55,075 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:04:55,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:04:55,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:04:56,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:04:58,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:04,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:06,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:10,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:05:10,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:05:10,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:10,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:05:10,529 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:05:10,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:05:10,530 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:05:10,936 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:05:12,937 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:05:15,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:16,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:20,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:25,078 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:05:25,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:05:25,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:05:26,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:26,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:31,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:36,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:37,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:40,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:05:40,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:05:42,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:46,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:47,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:52,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:05:55,080 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:05:55,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:05:55,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:05:56,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:05:58,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:03,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:06,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:06:09,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:10,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:06:10,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:06:15,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:16,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:06:21,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:25,082 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:06:25,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:06:25,153 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:06:26,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:06:27,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:32,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:36,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:06:38,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:40,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:06:40,154 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:06:43,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:46,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:06:48,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:54,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:06:55,085 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:06:55,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:06:55,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:06:56,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:00,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:06,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:06,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:10,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:07:10,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:07:12,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:13,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:07:13,561 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:07:13,561 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:07:13,562 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:07:14,012 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:07:15,012 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:07:16,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:17,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:23,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:25,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:07:25,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:07:25,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:07:26,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:28,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:33,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:36,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:39,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:40,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:07:40,154 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:07:44,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:46,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:50,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:07:55,089 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:07:55,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:07:55,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:07:56,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:07:56,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:01,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:06,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:08:06,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:10,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:08:10,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:08:12,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:16,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:08:18,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:23,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:25,093 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:08:25,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:08:25,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:08:26,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:08:28,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:34,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:36,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:08:39,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:40,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:08:40,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:08:45,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:46,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:08:50,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:55,094 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:08:55,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:08:55,155 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:08:55,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:08:56,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:00,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:05,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:06,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:10,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:09:10,156 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:09:11,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:14,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:09:14,950 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:09:14,950 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:09:14,951 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:09:15,079 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:09:16,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:16,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:17,080 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:09:22,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:25,097 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:09:25,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:09:25,156 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:09:26,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:28,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:33,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:36,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:38,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:40,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:09:40,156 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:09:44,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:46,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:49,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:54,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:09:55,099 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:09:55,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:09:55,157 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:09:56,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:09:59,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:04,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:06,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:10:10,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:10:10,156 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:10:10,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:15,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:16,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:10:20,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:25,101 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:10:25,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:10:25,157 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:10:25,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:26,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:10:31,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:36,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:10:37,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:40,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:10:40,157 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:10:43,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:46,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:10:49,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:55,104 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:10:55,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:10:55,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:10:55,157 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:10:56,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:00,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:05,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:06,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:10,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:11:10,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:11:11,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:16,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:16,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:11:16,205 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:11:16,206 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:11:16,206 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:11:17,142 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:11:17,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:18,142 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:11:22,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:25,106 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:11:25,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:11:25,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:11:26,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:28,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:34,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:11:36,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:40,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:11:40,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:11:46,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:11:51,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: keepalive 2024-08-30 21:11:56,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:01,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: keepalive 2024-08-30 21:12:01,534 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:12:01,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:06,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:07,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:10,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:12:10,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:12:13,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:16,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:18,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:23,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:25,114 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:12:25,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:12:25,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:12:26,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:29,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:34,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:36,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:40,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:12:40,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:12:40,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:46,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:46,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:52,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:12:55,113 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:12:55,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:12:55,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:12:56,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:12:57,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:03,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:06,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:13:08,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:10,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:13:10,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:13:14,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:16,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:13:19,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:13:19,022 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:13:19,022 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:13:19,023 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:13:19,199 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:13:20,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:20,199 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:13:25,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:13:25,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:25,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:13:25,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:13:26,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:13:30,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:35,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:36,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:13:40,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:40,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:13:40,316 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:13:45,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:46,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:13:50,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:55,118 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:13:55,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:13:55,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:13:55,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:13:56,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:01,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:06,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:07,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:10,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:14:10,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:14:12,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:16,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:17,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:23,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:25,120 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:14:25,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:14:25,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:14:26,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:28,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:34,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:36,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:39,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:40,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:14:40,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:14:45,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:46,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:51,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:14:55,123 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:14:55,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:14:55,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:14:56,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:14:56,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:02,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:06,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:07,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:10,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:15:10,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:15:13,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:16,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:19,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:20,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:15:20,395 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:15:20,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:15:20,396 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:15:21,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:15:22,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:15:25,126 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:15:25,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:25,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:15:25,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:15:26,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:30,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:36,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:36,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:40,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:15:40,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:15:41,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:46,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:47,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:52,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:15:55,128 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:15:55,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:15:55,300 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:15:56,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:15:58,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:04,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:06,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:16:10,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:10,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:16:10,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:16:15,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:16,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:16:21,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:25,130 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:16:25,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:16:25,316 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:16:26,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:16:26,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:32,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:36,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:16:38,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:40,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:16:40,316 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:16:43,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:46,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:16:49,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:54,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:16:55,136 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:16:55,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:16:55,316 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:16:56,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:00,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:05,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:06,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:10,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:10,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:17:10,317 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:17:15,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:16,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:21,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:22,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:17:22,023 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:17:22,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:17:22,024 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:17:22,317 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:17:22,318 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:17:25,135 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:17:25,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:17:25,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:17:26,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:26,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:32,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:36,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:37,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:40,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:17:40,318 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:17:42,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:46,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:48,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:54,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:17:55,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:17:55,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:17:55,318 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:17:56,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:17:59,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:04,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:06,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:18:09,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:10,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:18:10,318 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:18:15,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:16,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:18:21,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:25,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:18:25,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:18:25,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:18:26,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:18:26,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:32,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:36,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:18:38,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:40,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:18:40,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:18:44,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:46,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:18:50,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:55,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:18:55,142 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:18:55,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:18:55,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:18:56,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:00,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:06,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:06,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:10,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:19:10,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:19:11,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:16,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:17,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:23,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:25,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:19:25,140 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:19:25,141 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:19:25,145 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:19:25,146 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:19:25,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:19:25,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:19:25,373 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:19:26,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:26,374 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:19:29,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:34,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:36,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:39,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:40,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:19:40,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:19:44,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:46,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:19:50,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:55,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:19:55,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:19:55,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:19:55,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:19:56,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:01,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:06,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:06,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:10,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:20:10,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:20:11,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:16,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:17,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:22,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:25,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:20:25,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:20:25,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:20:26,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:27,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:33,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:36,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:38,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:40,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:20:40,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:20:44,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:46,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:20:50,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:55,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:20:55,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:20:55,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:20:55,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:20:56,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:00,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:06,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:06,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:10,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:21:10,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:21:11,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:16,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:17,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:22,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:25,159 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:21:25,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:21:25,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:21:26,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:26,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:21:26,548 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:21:26,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:21:26,550 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:21:27,428 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:21:27,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:28,428 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:21:33,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:36,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:38,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:40,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:21:40,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:21:43,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:46,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:21:49,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:55,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:21:55,156 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:21:55,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:21:55,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:21:56,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:01,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:06,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:06,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:10,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:22:10,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:22:11,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:16,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:17,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:23,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:25,161 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:22:25,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:22:25,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:22:26,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:28,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:33,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:36,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:39,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:40,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:22:40,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:22:45,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:46,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:51,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:22:55,161 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:22:55,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:22:55,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:22:56,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:22:56,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:02,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:06,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:23:07,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:10,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:23:10,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:23:12,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:16,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:23:18,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:24,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:25,163 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:23:25,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:23:25,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:23:26,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:23:28,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:23:28,174 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:23:28,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:23:28,176 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:23:28,489 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:23:28,490 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:23:29,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:35,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:36,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:23:40,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:40,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:23:40,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:23:45,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:46,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:23:50,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:55,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:23:55,166 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:23:55,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:23:55,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:23:56,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:01,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:06,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:07,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:10,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:24:10,323 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:24:12,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:16,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:18,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:24,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:25,168 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:24:25,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:24:25,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:24:26,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:30,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:35,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:36,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:40,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:24:40,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:24:40,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:45,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:46,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:51,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:24:55,171 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:24:55,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:24:55,323 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:24:56,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:24:57,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:02,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:06,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:07,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:10,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:25:10,323 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:25:13,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:16,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:18,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:24,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:25,172 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:25:25,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:25:25,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:25:26,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:29,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:31,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:25:31,790 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:25:31,790 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:25:31,791 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:25:32,542 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:25:32,543 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:25:34,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:36,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:40,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:25:40,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:25:40,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:45,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:46,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:51,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:25:55,180 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:25:55,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:25:55,325 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:25:56,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:25:57,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:02,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:06,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:07,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:10,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:26:10,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:26:13,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:16,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:18,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:24,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:25,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:26:25,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:26:25,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:26:26,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:29,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:34,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:36,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:39,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:40,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:26:40,325 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:26:45,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:46,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:50,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:26:55,182 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:26:55,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:26:55,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:26:56,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:26:56,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:01,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:06,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:07,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:10,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:27:10,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:27:13,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:16,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:18,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:24,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:25,183 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:27:25,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:27:25,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:27:26,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:30,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:33,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:27:33,459 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:27:33,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:27:33,463 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:27:33,599 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:27:34,599 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:27:35,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:36,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:40,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:27:40,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:27:41,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:46,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:46,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:52,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:27:55,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:27:55,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:27:55,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:27:56,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:27:58,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:03,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:06,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:08,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:10,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:28:10,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:28:14,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:16,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:19,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:25,188 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:28:25,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:25,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:28:25,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:28:26,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:30,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:36,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:36,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:40,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:28:40,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:28:42,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:46,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:47,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:53,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:28:55,190 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:28:55,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:28:55,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:28:56,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:28:58,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:03,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:06,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:08,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:10,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:29:10,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:29:13,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:16,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:19,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:24,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:25,193 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:29:25,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:29:25,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:29:26,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:30,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:35,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:29:35,111 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:29:35,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:29:35,112 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:29:35,664 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:29:36,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:36,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:36,664 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:29:40,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:29:40,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:29:41,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:46,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:46,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:52,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:29:55,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:29:55,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:29:55,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:29:56,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:29:58,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:03,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:06,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:08,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:10,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:30:10,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:30:13,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:16,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:18,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:23,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:25,198 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:30:25,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:30:25,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:30:26,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:29,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:35,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:36,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:40,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:30:40,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:30:41,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:46,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:46,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:52,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:30:55,205 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:30:55,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:30:55,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:30:56,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:30:58,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:03,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:06,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:08,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:10,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:31:10,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:31:14,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:16,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:20,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:25,204 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:31:25,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:25,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:31:25,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:31:26,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:31,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:36,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:37,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:38,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:31:38,574 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:31:38,574 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:31:38,576 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:31:38,721 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:31:40,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:31:40,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:31:40,722 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:31:42,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:46,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:48,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:54,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:31:55,206 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:31:55,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:31:55,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:31:56,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:31:59,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:05,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:06,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:10,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:32:10,353 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:32:10,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:16,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:16,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:21,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:25,209 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:32:25,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:32:25,355 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:32:26,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:26,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:32,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:36,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:37,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:40,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:32:40,355 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:32:42,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:46,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:48,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:53,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:32:55,211 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:32:55,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:32:55,355 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:32:56,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:32:58,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:04,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:06,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:10,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:10,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:33:10,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:33:16,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:16,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:21,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:25,212 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:33:25,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:33:25,355 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:33:26,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:26,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:32,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:36,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:37,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:40,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:33:40,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:33:40,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:33:40,552 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:33:40,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:33:40,553 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:33:40,793 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:33:42,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:42,799 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:33:46,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:48,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:53,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:33:55,214 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:33:55,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:33:55,355 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:33:56,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:33:58,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:04,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:06,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:09,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:10,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:34:10,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:34:14,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:16,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:20,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:25,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:25,217 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:34:25,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:34:25,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:34:26,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:31,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:36,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:37,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:40,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:34:40,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:34:42,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:46,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:48,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:54,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:34:55,219 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:34:55,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:34:55,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:34:56,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:34:59,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:05,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:06,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:10,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:10,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:35:10,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:35:16,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:16,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:21,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:25,225 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:35:25,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:35:25,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:35:26,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:26,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:32,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:36,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:37,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:40,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:35:40,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:35:41,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:35:41,904 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:35:41,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:35:41,907 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:35:42,852 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:35:42,853 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:35:42,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:46,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:48,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:54,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:35:55,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:35:55,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:35:55,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:35:56,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:35:59,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:05,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:06,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:10,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:10,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:36:10,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:36:15,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:16,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:20,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:25,227 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:36:25,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:25,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:36:25,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:36:26,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:31,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:36,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:36,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:40,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:36:40,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:36:41,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:46,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:47,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:52,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:36:55,228 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:36:55,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:36:55,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:36:56,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:36:57,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:03,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:06,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:09,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:10,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:37:10,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:37:14,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:16,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:20,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:25,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:25,230 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:37:25,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:37:25,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:37:26,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:31,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:36,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:37,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:40,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:37:40,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:37:42,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:45,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:37:45,133 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:37:45,133 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:37:45,134 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:37:45,915 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:37:46,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:46,916 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:37:48,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:54,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:37:55,233 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:37:55,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:37:55,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:37:56,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:37:59,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:05,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:06,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:10,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:10,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:38:10,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:38:15,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:16,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:21,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:25,235 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:38:25,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:38:25,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:38:26,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:26,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:32,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:36,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:37,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:40,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:38:40,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:38:42,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:46,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:48,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:54,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:38:55,239 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:38:55,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:38:55,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:38:56,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:38:59,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:05,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:06,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:39:10,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:39:10,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:39:10,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:16,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:16,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:39:22,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:25,241 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:39:25,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:39:25,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:39:26,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:39:27,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:33,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:36,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:39:39,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:40,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:39:40,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:39:44,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:46,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:39:46,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:39:46,792 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:39:46,792 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:39:46,793 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:39:46,979 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:39:48,980 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:39:49,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:55,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:39:55,249 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:39:55,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:39:55,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:39:56,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:00,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:06,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:06,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:10,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:40:10,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:40:11,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:16,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:17,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:22,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:25,246 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:40:25,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:40:25,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:40:26,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:27,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:33,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:36,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:39,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:40,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:40:40,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:40:44,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:46,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:40:50,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:55,250 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:40:55,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:40:55,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:40:55,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:40:56,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:01,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:06,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:07,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:10,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:41:10,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:41:12,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:16,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:18,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:24,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:25,251 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:41:25,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:41:25,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:41:26,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:29,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:35,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:36,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:40,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:41:40,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:41:40,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:46,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:46,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:48,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:41:48,785 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:41:48,786 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:41:48,788 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:41:49,044 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:41:51,045 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:41:51,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:41:55,253 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:41:55,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:41:55,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:41:56,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:41:56,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:02,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:06,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:08,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:10,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:42:10,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:42:13,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:16,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:19,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:24,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:25,259 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:42:25,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:42:25,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:42:26,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:29,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:35,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:36,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:40,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:42:40,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:42:40,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:46,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:46,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:52,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:42:55,261 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:42:55,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:42:55,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:42:56,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:42:57,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:03,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:06,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:09,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:10,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:43:10,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:43:14,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:16,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:20,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:25,263 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:43:25,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:25,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:43:25,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:43:26,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:31,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:36,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:37,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:40,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:43:40,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:43:42,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:46,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:48,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:50,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:43:50,676 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:43:50,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:43:50,677 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:43:51,115 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:43:53,116 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:43:53,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:43:55,265 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:43:55,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:43:55,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:43:56,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:43:59,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:05,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:06,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:44:10,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:10,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:44:10,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:44:16,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:16,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:44:22,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:25,271 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:44:25,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:44:25,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:44:26,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:44:27,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:33,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:36,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:44:39,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:40,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:44:40,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:44:44,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:46,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:44:50,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:55,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:44:55,270 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:44:55,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:44:55,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:44:56,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:00,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:06,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:06,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:10,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:45:10,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:45:11,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:16,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:17,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:23,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:25,274 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:45:25,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:45:25,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:45:26,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:28,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:34,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:36,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:40,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:40,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:45:40,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:45:46,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:46,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:51,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:45:53,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:45:53,497 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:45:53,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:45:53,498 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:45:54,167 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:45:55,168 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:45:55,275 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:45:55,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:45:55,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:45:56,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:45:56,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:02,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:06,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:08,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:10,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:46:10,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:46:13,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:16,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:19,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:25,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:25,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:46:25,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:46:25,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:46:26,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:30,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:36,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:36,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:40,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:46:40,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:46:41,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:46,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:47,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:52,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:46:55,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:46:55,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:46:55,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:46:56,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:46:57,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:03,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:06,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:09,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:10,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:47:10,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:47:14,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:16,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:20,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:25,282 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:47:25,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:25,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:47:25,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:47:26,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:31,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:36,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:37,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:40,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:47:40,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:47:42,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:46,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:48,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:54,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:47:55,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:47:55,087 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:47:55,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:47:55,088 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:47:55,215 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:47:55,284 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:47:55,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:47:55,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:47:56,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:47:57,217 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:47:59,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:05,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:06,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:10,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:48:10,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:48:10,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:16,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:16,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:21,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:25,286 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:48:25,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:48:25,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:48:26,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:32,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:36,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:37,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:40,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:48:40,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:48:42,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:46,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:48,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:54,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:48:55,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:48:55,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:48:55,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:48:56,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:48:59,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:05,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:06,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:10,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:49:10,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:49:10,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:16,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:16,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:21,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:25,291 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:49:25,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:49:25,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:49:26,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:26,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:32,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:36,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:37,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:40,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:49:40,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:49:42,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:46,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:48,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:54,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:49:55,296 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:49:55,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:49:55,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:49:56,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:49:56,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:49:56,604 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:49:56,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:49:56,605 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:49:57,267 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:49:59,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:49:59,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:05,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:06,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:10,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:50:10,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:50:10,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:16,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:16,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:22,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:25,296 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:50:25,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:50:25,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:50:26,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:27,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:33,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:36,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:39,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:40,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:50:40,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:50:45,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:46,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:51,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:50:55,298 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:50:55,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:50:55,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:50:56,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:50:57,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:02,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:06,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:07,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:10,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:51:10,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:51:12,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:16,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:17,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:23,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:25,301 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:51:25,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:51:25,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:51:26,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:29,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:34,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:36,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:39,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:40,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:51:40,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:51:45,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:46,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:51,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:55,304 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:51:55,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:51:55,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:51:56,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:51:57,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:51:59,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:51:59,920 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:51:59,920 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:51:59,921 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:52:00,325 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:52:01,325 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:52:02,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:06,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:08,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:10,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:52:10,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:52:13,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:16,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:19,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:25,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:25,306 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:52:25,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:52:25,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:52:26,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:30,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:36,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:36,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:40,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:52:40,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:52:42,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:46,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:47,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:53,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:52:55,308 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:52:55,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:52:55,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:52:56,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:52:58,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:04,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:06,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:10,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:10,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:53:10,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:53:15,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:16,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:20,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:25,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:25,310 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:53:25,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:53:25,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:53:26,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:31,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:36,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:37,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:40,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:53:40,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:53:43,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:46,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:48,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:54,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:53:55,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:53:55,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:53:55,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:53:56,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:53:59,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:01,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:54:01,544 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:54:01,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:54:01,546 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:54:02,388 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:54:03,389 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:54:04,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:06,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:10,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:10,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:54:10,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:54:16,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:16,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:21,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:25,315 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:54:25,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:54:25,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:54:26,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:26,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:31,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:36,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:37,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:40,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:54:40,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:54:42,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:46,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:47,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:53,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:54:55,319 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:54:55,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:54:55,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:54:56,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:54:58,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:03,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:06,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:09,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:10,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:55:10,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:55:14,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:16,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:20,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:25,320 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:55:25,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:25,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:55:25,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:55:26,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:31,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:36,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:37,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:40,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:55:40,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:55:42,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:46,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:47,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:53,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:55:55,322 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:55:55,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:55:55,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:55:56,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:55:58,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:03,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:03,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:56:03,380 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:56:03,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:56:03,381 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:56:03,447 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:56:05,448 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:56:06,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:09,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:10,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:56:10,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:56:14,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:16,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:19,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:25,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:25,324 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:56:25,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:56:25,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:56:26,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:31,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:36,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:36,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:40,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:56:40,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:56:42,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:46,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:47,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:53,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:56:55,327 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:56:55,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:56:55,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:56:56,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:56:58,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:04,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:06,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:57:10,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:10,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:57:10,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:57:15,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:16,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:57:21,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:25,330 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:57:25,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:57:25,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:57:26,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:57:27,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:32,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:36,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:57:38,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:40,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:57:40,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:57:44,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:46,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:57:49,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:54,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:57:55,332 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:57:55,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:57:55,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:57:56,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:00,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:05,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:06,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:06,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 21:58:06,958 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 21:58:06,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 21:58:06,959 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 21:58:07,504 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 21:58:09,504 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 21:58:10,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:58:10,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:58:10,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:16,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:16,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:22,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:25,339 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:58:25,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:58:25,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:58:26,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:28,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:33,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:36,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:39,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:40,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:58:40,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:58:45,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:46,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:58:50,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:55,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:58:55,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:58:55,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:58:55,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:58:56,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:01,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:06,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:06,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:10,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:59:10,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:59:11,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:16,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:17,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:22,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:25,341 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:59:25,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:59:25,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:59:26,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:27,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:32,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:36,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:38,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:40,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:59:40,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:59:44,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:46,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 21:59:50,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:55,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 21:59:55,341 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 21:59:55,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 21:59:55,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 21:59:56,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:01,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:06,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:06,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:09,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:00:09,385 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:00:09,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:00:09,386 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:00:09,573 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:00:10,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:00:10,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:00:11,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:11,574 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:00:16,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:17,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:23,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:25,343 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:00:25,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:00:25,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:00:26,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:28,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:33,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:36,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:38,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:40,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:00:40,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:00:44,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:46,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:00:49,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:55,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:00:55,345 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:00:55,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:00:55,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:00:56,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:01,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:06,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:07,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:10,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:01:10,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:01:13,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:16,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:18,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:23,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:25,348 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:01:25,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:01:25,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:01:26,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:28,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:34,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:36,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:39,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:40,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:01:40,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:01:45,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:46,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:01:50,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:55,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:01:55,350 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:01:55,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:01:55,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:01:56,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:00,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:06,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:06,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:10,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:02:10,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:02:11,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:02:11,502 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:02:11,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:02:11,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:11,504 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:02:11,635 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:02:13,636 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:02:16,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:17,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:23,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:25,355 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:02:25,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:02:25,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:02:26,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:28,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:34,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:36,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:39,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:40,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:02:40,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:02:45,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:46,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:02:50,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:55,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:02:55,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:02:55,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:02:55,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:02:56,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:01,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:06,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:06,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:10,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:03:10,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:03:12,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:16,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:17,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:23,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:25,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:03:25,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:03:25,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:03:26,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:29,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:35,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:36,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:40,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:40,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:03:40,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:03:45,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:46,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:51,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:03:55,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:03:55,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:03:55,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:03:56,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:03:56,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:02,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:06,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:07,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:10,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:04:10,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:04:12,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:14,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:04:14,900 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:04:14,900 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:04:14,901 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:04:15,704 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:04:16,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:17,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:17,705 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:04:23,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:25,366 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:04:25,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:04:25,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:04:26,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:28,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:34,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:36,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:39,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:40,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:04:40,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:04:45,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:46,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:51,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:04:55,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:04:55,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:04:55,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:04:56,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:04:57,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:02,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:06,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:07,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:10,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:05:10,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:05:13,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:16,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:19,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:25,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:25,370 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:05:25,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:05:25,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:05:26,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:30,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:36,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:36,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:40,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:05:40,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:05:41,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:46,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:46,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:52,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:05:55,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:05:55,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:05:55,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:05:56,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:05:58,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:03,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:06,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:08,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:10,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:06:10,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:06:14,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:16,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:16,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:06:16,603 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:06:16,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:06:16,604 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:06:16,765 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:06:19,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:19,766 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:06:25,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:25,376 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:06:25,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:06:25,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:06:26,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:31,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:36,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:37,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:40,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:06:40,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:06:42,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:46,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:48,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:53,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:06:55,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:06:55,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:06:55,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:06:56,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:06:59,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:04,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:06,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:10,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:10,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:07:10,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:07:15,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:16,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:21,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:25,386 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:07:25,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:07:25,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:07:26,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:26,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:31,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:36,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:37,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:40,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:07:40,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:07:42,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:46,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:47,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:52,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:07:55,384 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:07:55,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:07:55,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:07:56,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:07:58,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:03,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:06,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:08:09,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:10,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:08:10,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:08:14,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:16,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:08:18,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:08:18,618 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:08:18,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:08:18,621 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:08:18,812 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:08:19,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:21,813 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:08:25,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:25,387 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:08:25,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:08:25,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:08:26,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:08:31,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:36,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:08:37,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:40,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:08:40,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:08:42,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:46,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:08:48,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:54,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:08:55,391 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:08:55,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:08:55,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:08:56,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:00,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:05,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:06,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:10,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:09:10,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:09:10,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:16,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:16,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:21,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:25,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:09:25,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:09:25,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:09:26,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:27,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:32,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:36,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:38,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:40,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:09:40,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:09:43,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:46,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:09:49,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:54,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:09:55,395 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:09:55,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:09:55,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:09:56,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:00,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:05,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:06,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:10,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:10:10,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:10:10,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:16,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:16,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:21,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:21,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:10:21,836 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:10:21,837 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:10:21,838 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:10:21,858 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:10:23,859 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:10:25,399 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:10:25,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:10:25,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:10:26,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:26,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:32,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:36,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:38,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:40,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:10:40,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:10:43,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:46,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:48,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:53,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:10:55,400 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:10:55,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:10:55,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:10:56,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:10:59,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:04,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:06,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:09,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:10,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:11:10,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:11:14,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:16,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:19,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:25,404 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:11:25,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:25,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:11:25,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:11:26,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:30,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:35,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:36,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:40,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:11:40,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:11:40,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:46,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:46,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:52,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:11:55,407 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:11:55,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:11:55,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:11:56,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:11:58,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:03,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:06,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:12:09,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:10,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:12:10,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:12:15,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:16,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:12:20,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:23,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:12:23,458 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:12:23,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:12:23,459 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:12:23,926 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:12:25,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:12:25,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:12:25,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:12:25,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:25,927 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:12:26,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:12:31,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:36,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:12:37,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:40,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:12:40,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:12:43,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:46,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:12:48,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:54,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:12:55,411 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:12:55,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:12:55,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:12:56,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:00,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:06,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:06,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:10,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:13:10,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:13:12,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:16,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:17,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:22,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:25,415 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:13:25,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:13:25,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:13:26,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:28,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:33,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:36,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:38,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:40,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:13:40,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:13:44,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:46,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:13:50,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:55,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:13:55,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:13:55,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:13:55,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:13:56,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:01,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:06,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:06,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:10,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:14:10,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:14:11,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:16,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:17,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:22,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:25,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:14:25,063 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:14:25,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:14:25,064 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:14:25,418 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:14:25,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:14:25,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:14:25,976 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:14:26,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:27,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:27,977 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:14:33,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:36,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:39,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:40,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:14:40,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:14:45,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:46,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:14:50,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:55,422 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:14:55,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:14:55,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:14:55,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:14:56,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:01,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:06,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:06,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:10,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:15:10,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:15:11,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:16,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:17,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:22,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:25,425 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:15:25,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:15:25,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:15:26,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:28,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:34,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:36,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:39,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:40,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:15:40,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:15:45,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:46,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:51,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:15:55,427 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:15:55,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:15:55,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:15:56,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:15:56,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:02,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:06,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:08,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:10,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:16:10,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:16:13,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:16,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:18,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:24,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:25,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:16:25,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:16:25,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:16:26,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:26,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:16:26,526 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:16:26,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:16:26,527 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:16:27,032 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:16:27,032 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:16:30,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:35,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:36,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:40,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:16:40,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:16:41,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:46,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:46,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:52,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:16:55,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:16:55,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:16:55,636 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:16:56,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:16:57,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:02,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:06,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:08,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:10,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:17:10,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:17:14,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:16,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:19,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:25,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:17:25,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:17:25,608 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:17:25,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:26,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:31,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:36,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:37,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:40,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:17:40,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:17:43,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:46,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:48,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:54,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:17:55,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:17:55,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:17:55,635 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:17:56,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:17:59,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:04,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:06,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:10,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:18:10,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:18:10,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:16,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:16,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:21,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:25,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:18:25,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:18:25,623 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:18:26,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:26,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:29,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:18:29,602 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:18:29,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:18:29,603 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:18:30,078 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:18:31,078 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:18:31,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:36,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:37,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:40,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:18:40,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:18:42,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:46,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:48,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:53,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:18:55,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:18:55,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:18:55,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:18:56,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:18:58,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:04,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:06,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:10,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:19:10,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:19:10,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:16,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:16,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:21,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:25,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:19:25,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:19:25,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:19:26,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:27,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:32,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:36,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:37,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:40,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:19:40,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:19:42,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:46,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:48,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:53,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:19:55,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:19:55,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:19:55,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:19:56,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:19:58,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:04,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:06,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:09,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:10,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:20:10,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:20:15,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:16,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:20,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:25,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:20:25,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:20:25,558 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:20:26,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:26,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:31,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:20:31,306 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:20:31,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:20:31,307 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:20:32,140 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:20:32,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:33,140 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:20:36,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:37,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:40,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:20:40,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:20:43,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:46,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:48,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:53,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:20:55,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:20:55,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:20:55,562 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:20:56,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:20:59,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:04,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:06,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:10,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:21:10,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:21:10,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:16,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:16,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:21,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:25,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:21:25,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:21:25,633 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:21:26,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:26,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:32,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:36,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:37,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:40,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:21:40,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:21:43,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:46,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:48,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:53,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:21:55,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:21:55,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:21:55,564 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:21:56,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:21:59,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:04,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:06,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:22:10,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:22:10,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:22:10,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:15,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:16,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:22:21,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:25,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:22:25,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:22:25,604 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:22:26,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:22:27,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:32,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:32,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:22:32,902 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:22:32,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:22:32,903 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:22:33,188 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:22:35,190 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:22:36,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:22:38,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:40,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:22:40,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:22:43,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:46,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:22:49,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:55,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:22:55,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:22:55,593 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:22:55,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:22:56,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:01,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:06,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:07,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:10,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:23:10,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:23:12,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:16,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:18,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:23,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:25,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:23:25,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:23:25,599 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:23:26,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:28,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:34,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:36,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:40,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:40,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:23:40,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:23:46,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:46,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:51,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:23:55,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:23:55,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:23:55,657 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:23:56,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:23:56,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:02,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:06,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:07,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:10,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:24:10,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:24:12,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:16,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:18,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:23,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:25,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:24:25,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:24:25,596 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:24:26,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:29,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:34,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:35,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:24:35,878 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:24:35,879 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:24:35,882 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:24:36,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:36,242 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:24:37,242 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:24:40,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:40,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:24:40,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:24:46,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:46,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:51,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:24:55,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:24:55,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:24:55,649 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:24:56,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:24:56,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:02,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:06,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:07,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:10,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:25:10,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:25:12,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:16,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:18,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:23,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:25,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:25:25,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:25:25,656 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:25:26,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:29,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:34,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:36,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:39,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:40,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:25:40,441 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:25:45,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:46,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:51,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:25:55,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:25:55,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:25:55,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:25:56,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:25:56,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:02,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:06,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:07,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:10,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:26:10,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:26:12,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:16,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:18,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:24,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:25,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:26:25,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:26:25,625 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:26:26,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:29,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:35,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:36,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:37,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:26:37,551 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:26:37,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:26:37,552 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:26:38,289 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:26:39,289 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:26:40,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:40,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:26:40,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:26:46,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:46,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:52,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:26:55,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:26:55,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:26:55,588 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:26:56,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:26:57,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:03,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:06,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:27:09,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:10,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:27:10,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:27:15,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:16,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:27:21,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:25,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:27:25,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:27:25,604 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:27:26,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:27:26,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:32,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:36,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:27:38,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:40,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:27:40,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:27:44,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:46,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:27:50,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:55,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:27:55,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:27:55,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:27:55,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:27:56,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:01,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:06,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:07,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:10,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:28:10,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:28:12,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:16,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:18,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:24,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:25,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:28:25,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:28:25,590 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:28:26,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:29,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:35,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:36,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:39,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:28:39,013 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:28:39,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:28:39,015 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:28:39,343 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:28:40,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:28:40,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:28:40,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:41,343 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:28:46,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:46,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:52,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:28:55,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:28:55,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:28:55,662 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:28:56,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:28:57,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:03,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:06,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:08,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:10,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:29:10,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:29:13,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:16,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:18,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:23,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:25,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:29:25,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:29:25,588 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:29:26,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:28,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:34,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:36,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:39,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:40,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:29:40,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:29:45,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:46,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:51,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:29:55,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:29:55,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:29:55,642 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:29:56,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:29:56,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:02,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:06,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:30:08,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:10,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:30:10,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:30:13,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:16,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:30:19,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:24,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:25,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:30:25,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:30:25,602 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:30:26,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:30:29,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:35,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:36,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:30:40,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:40,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:30:40,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:30:42,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:30:42,209 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:30:42,210 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:30:42,212 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:30:42,390 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:30:43,391 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:30:45,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:46,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:30:50,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:55,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:30:55,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:30:55,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:30:55,599 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:30:56,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:01,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:06,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:07,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:10,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:31:10,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:31:12,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:16,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:18,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:24,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:25,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:31:25,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:31:25,671 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:31:26,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:30,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:36,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:36,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:40,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:31:40,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:31:41,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:46,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:47,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:52,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:31:55,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:31:55,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:31:55,634 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:31:56,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:31:57,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:03,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:06,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:08,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:10,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:32:10,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:32:14,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:16,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:19,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:24,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:25,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:32:25,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:32:25,648 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:32:26,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:30,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:36,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:36,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:40,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:32:40,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:32:41,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:43,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:32:43,846 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:32:43,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:32:43,847 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:32:44,437 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:32:45,438 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:32:46,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:46,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:52,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:32:55,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:32:55,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:32:55,637 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:32:56,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:32:57,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:03,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:06,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:09,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:10,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:33:10,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:33:15,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:16,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:20,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:25,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:33:25,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:33:25,665 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:33:25,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:26,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:31,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:36,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:37,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:40,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:33:40,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:33:42,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:46,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:48,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:53,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:33:55,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:33:55,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:33:55,676 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:33:56,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:33:58,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:04,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:06,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:10,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:10,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:34:10,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:34:15,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:16,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:21,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:25,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:34:25,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:34:25,613 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:34:26,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:26,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:32,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:36,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:37,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:40,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:34:40,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:34:42,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:45,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:34:45,354 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:34:45,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:34:45,357 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:34:45,482 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:34:46,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:47,483 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:34:48,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:54,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:34:55,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:34:55,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:34:55,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:34:56,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:34:59,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:04,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:06,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:10,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:10,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:35:10,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:35:16,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:16,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:21,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:25,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:35:25,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:35:25,680 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:35:26,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:26,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:32,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:36,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:37,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:40,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:35:40,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:35:42,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:46,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:47,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:53,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:35:55,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:35:55,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:35:55,619 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:35:56,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:35:59,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:04,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:06,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:10,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:36:10,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:36:10,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:15,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:16,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:20,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:25,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:36:25,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:36:25,623 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:36:26,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:26,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:31,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:36,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:37,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:40,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:36:40,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:36:43,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:46,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:48,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:36:48,557 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:36:48,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:36:48,558 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:36:48,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:49,523 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:36:49,523 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:36:53,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:36:55,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:36:55,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:36:55,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:36:56,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:36:59,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:04,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:06,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:10,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:37:10,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:37:10,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:15,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:16,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:21,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:25,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:37:25,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:37:25,627 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:37:26,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:27,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:32,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:36,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:38,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:40,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:37:40,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:37:44,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:46,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:49,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:54,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:37:55,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:37:55,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:37:55,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:37:56,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:37:59,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:04,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:06,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:09,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:10,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:38:10,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:38:15,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:16,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:20,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:25,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:38:25,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:38:25,634 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:38:26,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:26,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:31,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:36,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:37,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:40,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:38:40,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:38:43,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:46,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:48,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:49,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:38:50,000 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:38:50,000 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:38:50,001 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:38:50,573 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:38:51,573 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:38:54,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:38:55,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:38:55,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:38:55,643 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:38:56,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:38:59,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:04,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:06,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:10,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:39:10,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:39:10,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:15,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:16,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:20,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:25,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:39:25,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:39:25,642 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:39:26,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:26,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:31,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:36,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:37,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:40,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:39:40,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:39:43,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:46,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:48,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:53,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:39:55,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:39:55,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:39:55,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:39:56,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:39:59,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:04,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:06,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:10,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:40:10,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:40:10,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:15,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:16,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:20,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:25,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:40:25,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:40:25,643 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:40:26,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:26,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:31,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:36,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:37,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:40,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:40:40,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:40:43,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:46,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:48,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:51,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:40:51,503 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:40:51,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:40:51,504 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:40:51,625 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:40:53,626 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:40:54,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:40:55,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:40:55,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:40:55,668 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:40:56,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:40:59,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:04,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:06,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:10,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:41:10,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:41:10,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:15,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:16,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:20,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:25,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:41:25,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:41:25,648 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:41:26,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:26,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:31,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:36,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:37,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:40,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:41:40,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:41:43,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:46,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:48,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:54,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:41:55,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:41:55,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:41:55,652 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:41:56,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:41:59,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:05,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:06,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:42:10,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:42:10,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:42:11,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:16,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:42:16,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:21,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:25,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:42:25,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:42:25,654 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:42:26,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:42:27,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:33,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:36,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:42:38,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:40,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:42:40,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:42:44,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:46,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:42:49,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:54,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:42:54,661 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:42:54,662 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:42:54,662 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:42:54,678 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:42:55,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:42:55,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:42:55,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:42:55,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:42:55,679 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:42:56,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:00,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:06,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:06,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:10,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:43:10,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:43:12,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:16,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:18,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:23,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:25,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:43:25,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:43:25,659 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:43:26,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:29,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:35,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:36,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:40,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:43:40,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:43:41,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:46,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:46,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:52,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:43:55,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:43:55,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:43:55,662 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:43:56,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:43:58,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:03,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:06,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:09,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:10,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:44:10,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:44:15,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:16,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:21,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:25,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:44:25,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:44:25,667 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:44:26,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:26,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:31,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:36,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:36,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:40,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:44:40,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:44:41,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:46,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:47,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:52,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:44:55,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:44:55,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:44:55,665 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:44:56,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:44:56,170 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:44:56,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:44:56,173 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:44:56,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:44:56,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:44:57,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:44:58,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:03,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:06,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:09,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:10,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:45:10,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:45:15,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:16,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:20,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:25,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:45:25,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:45:25,668 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:45:25,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:26,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:30,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:35,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:36,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:40,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:45:40,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:45:41,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:46,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:46,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:52,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:45:55,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:45:55,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:45:55,672 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:45:56,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:45:58,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:04,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:06,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:10,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:46:10,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:46:10,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:15,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:16,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:20,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:25,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:46:25,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:46:25,673 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:46:26,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:26,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:31,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:36,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:37,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:40,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:46:40,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:46:43,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:46,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:49,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:54,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:46:55,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:46:55,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:46:55,675 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:46:56,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:46:57,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:46:57,718 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:46:57,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:46:57,718 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:46:57,778 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:46:59,779 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:47:00,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:05,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:06,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:47:10,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:47:10,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:47:11,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:16,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:47:16,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:22,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:25,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:47:25,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:47:25,678 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:47:26,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:47:28,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:34,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:36,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:47:39,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:40,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:47:40,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:47:45,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:46,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:47:50,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:55,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:47:55,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:47:55,689 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:47:55,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:47:56,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:00,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:06,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:06,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:10,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:48:10,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:48:12,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:16,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:17,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:23,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:25,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:48:25,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:48:25,682 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:48:26,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:29,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:34,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:36,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:40,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:40,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:48:40,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:48:46,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:46,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:52,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:48:55,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:48:55,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:48:55,686 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:48:56,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:48:57,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:00,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:49:00,873 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:49:00,873 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:49:00,874 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:49:01,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:49:01,831 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:49:02,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:06,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:07,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:10,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:49:10,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:49:13,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:16,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:18,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:23,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:25,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:49:25,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:49:25,691 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:49:26,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:29,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:35,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:36,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:40,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:40,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:49:40,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:49:46,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:46,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:51,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:49:55,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:49:55,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:49:55,690 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:49:56,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:49:57,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:02,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:06,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:07,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:10,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:50:10,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:50:13,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:16,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:18,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:24,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:25,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:50:25,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:50:25,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:50:26,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:30,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:36,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:36,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:40,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:50:40,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:50:41,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:46,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:47,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:52,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:50:55,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:50:55,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:50:55,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:50:56,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:50:58,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:02,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:51:02,501 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:51:02,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:51:02,502 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:51:02,896 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:51:03,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:03,897 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:51:06,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:09,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:10,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:51:10,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:51:15,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:16,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:20,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:25,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:51:25,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:51:25,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:25,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:51:26,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:31,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:36,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:36,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:40,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:51:40,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:51:42,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:46,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:47,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:52,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:51:55,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:51:55,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:51:55,700 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:51:56,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:51:58,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:03,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:06,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:52:09,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:10,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:52:10,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:52:15,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:16,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:52:20,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:25,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:52:25,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:52:25,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:25,702 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:52:26,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:52:31,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:36,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:52:37,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:40,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:52:40,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:52:42,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:46,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:52:48,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:54,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:52:55,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:52:55,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:52:55,704 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:52:56,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:00,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:03,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:53:03,995 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:53:03,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:53:03,996 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:53:04,966 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:53:05,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:05,967 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:53:06,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:10,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:53:10,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:53:10,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:16,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:16,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:22,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:25,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:53:25,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:53:25,707 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:53:26,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:28,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:33,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:36,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:39,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:40,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:53:40,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:53:44,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:46,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:53:50,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:55,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:53:55,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:53:55,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:53:55,713 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:53:56,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:01,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:06,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:07,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:10,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:54:10,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:54:12,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:16,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:18,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:23,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:25,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:54:25,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:54:25,711 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:54:26,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:28,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:33,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:36,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:39,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:40,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:54:40,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:54:44,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:46,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:54:50,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:55,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:54:55,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:54:55,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:54:55,716 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:54:56,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:01,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:05,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:55:05,665 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:55:05,665 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:55:05,666 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:55:06,026 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:55:06,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:07,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:08,027 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:55:11,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:55:11,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:55:12,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:16,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:18,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:24,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:25,716 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:55:26,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:55:26,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:55:26,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:29,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:35,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:36,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:40,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:41,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:55:41,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:55:45,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:46,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:55:51,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:55,719 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:55:56,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:55:56,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:55:56,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:55:56,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:02,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:06,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:08,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:11,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:56:11,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:56:13,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:16,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:19,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:25,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:25,721 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:56:26,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:56:26,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:56:26,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:30,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:36,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:36,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:41,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:56:41,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:56:41,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:46,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:47,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:53,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:56:55,724 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:56:56,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:56:56,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:56:56,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:56:58,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:04,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:06,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:08,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:57:08,810 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:57:08,810 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:57:08,811 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:57:09,079 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:57:09,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:10,080 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:57:11,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:57:11,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:57:15,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:16,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:21,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:25,727 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:57:26,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:57:26,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:57:26,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:26,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:32,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:36,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:38,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:41,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:57:41,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:57:43,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:46,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:49,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:54,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:57:55,729 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:57:56,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:57:56,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:57:56,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:57:59,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:05,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:06,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:58:11,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:58:11,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:58:11,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:16,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:58:17,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:22,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:25,736 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:58:26,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:58:26,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:58:26,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:58:27,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:33,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:36,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:58:38,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:41,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:58:41,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:58:43,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:46,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:58:49,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:55,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:58:55,734 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:58:56,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:58:56,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:58:56,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:00,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:06,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:06,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:10,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 22:59:10,361 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 22:59:10,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 22:59:10,363 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 22:59:11,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:59:11,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:59:11,139 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 22:59:11,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:12,139 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 22:59:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:17,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:22,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:25,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:59:26,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:59:26,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:59:26,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:27,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:33,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:36,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:38,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:41,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:59:41,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:59:43,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:46,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:49,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:54,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 22:59:55,741 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 22:59:56,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 22:59:56,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 22:59:56,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 22:59:59,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:05,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:06,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:00:11,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:00:11,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:00:11,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:00:17,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:23,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:25,742 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:00:26,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:00:26,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:00:26,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:00:28,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:34,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:36,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:00:39,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:41,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:00:41,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:00:44,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:46,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:00:50,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:55,744 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:00:55,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:00:56,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:00:56,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:00:56,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:01,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:06,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:06,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:11,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:01:11,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:01:11,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:11,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:01:11,919 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:01:11,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:01:11,920 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:01:12,191 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:01:14,192 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:01:16,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:17,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:22,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:25,747 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:01:26,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:01:26,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:01:26,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:27,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:33,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:36,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:39,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:41,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:01:41,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:01:44,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:46,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:01:50,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:55,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:01:55,751 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:01:56,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:01:56,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:01:56,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:00,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:06,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:06,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:11,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:02:11,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:02:11,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:16,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:17,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:22,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:25,754 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:02:26,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:02:26,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:02:26,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:27,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:33,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:36,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:39,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:41,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:02:41,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:02:44,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:46,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:02:50,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:55,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:02:55,756 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:02:56,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:02:56,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:02:56,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:00,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:06,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:06,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:11,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:03:11,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:03:11,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:14,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:03:14,980 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:03:14,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:03:14,981 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:03:15,245 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:03:16,246 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:03:16,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:17,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:22,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:25,758 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:03:26,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:03:26,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:03:26,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:27,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:33,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:36,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:39,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:41,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:03:41,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:03:44,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:46,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:03:50,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:55,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:03:55,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:03:56,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:03:56,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:03:56,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:00,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:06,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:06,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:11,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:04:11,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:04:11,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:16,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:17,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:22,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:25,763 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:04:26,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:04:26,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:04:26,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:27,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:33,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:36,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:39,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:41,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:04:41,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:04:44,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:46,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:04:50,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:55,766 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:04:55,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:04:56,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:04:56,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:04:56,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:01,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:06,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:07,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:11,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:05:11,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:05:12,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:16,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:16,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:05:16,723 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:05:16,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:05:16,724 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:05:17,299 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:05:17,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:18,300 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:05:23,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:25,767 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:05:26,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:05:26,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:05:26,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:28,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:34,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:36,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:39,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:41,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:05:41,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:05:44,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:46,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:05:50,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:55,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:05:55,769 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:05:56,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:05:56,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:05:56,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:00,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:06,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:06,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:11,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:06:11,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:06:11,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:16,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:17,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:23,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:25,772 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:06:26,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:06:26,039 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:06:26,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:28,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:34,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:36,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:40,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:41,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:06:41,039 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:06:45,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:46,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:06:51,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:55,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:06:56,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:06:56,039 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:06:56,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:06:56,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:02,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:06,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:08,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:11,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:07:11,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:07:13,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:16,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:18,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:07:18,333 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:07:18,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:07:18,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:18,335 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:07:18,348 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:07:20,349 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:07:24,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:25,777 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:07:26,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:07:26,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:07:26,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:29,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:35,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:36,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:41,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:07:41,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:07:41,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:46,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:47,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:53,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:07:55,782 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:07:56,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:07:56,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:07:56,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:07:58,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:04,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:06,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:10,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:11,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:08:11,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:08:15,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:16,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:21,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:25,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:08:26,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:08:26,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:08:26,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:26,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:32,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:36,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:37,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:41,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:08:41,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:08:42,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:46,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:48,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:53,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:08:55,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:08:56,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:08:56,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:08:56,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:08:58,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:04,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:06,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:10,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:11,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:09:11,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:09:15,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:16,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:21,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:21,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:09:21,570 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:09:21,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:09:21,571 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:09:22,404 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:09:22,404 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:09:25,786 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:09:26,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:09:26,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:09:26,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:26,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:32,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:36,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:38,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:41,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:09:41,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:09:43,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:46,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:49,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:54,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:09:55,789 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:09:56,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:09:56,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:09:56,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:09:59,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:05,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:06,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:10:10,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:11,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:10:11,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:10:15,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:16,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:10:21,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:25,791 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:10:26,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:10:26,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:10:26,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:26,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:10:32,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:36,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:10:38,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:41,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:10:41,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:10:43,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:46,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:10:49,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:55,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:10:55,793 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:10:56,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:10:56,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:10:56,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:00,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:06,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:06,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:11,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:11:11,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:11:11,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:16,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:17,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:22,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:23,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:11:23,166 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:11:23,167 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:11:23,167 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:11:23,452 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:11:24,452 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:11:25,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:11:26,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:11:26,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:11:26,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:27,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:33,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:36,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:39,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:41,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:11:41,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:11:44,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:46,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:11:50,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:55,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:11:55,798 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:11:56,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:11:56,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:11:56,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:00,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:06,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:06,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:11,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:12:11,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:12:11,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:16,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:17,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:23,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:25,805 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:12:26,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:12:26,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:12:26,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:28,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:34,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:36,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:39,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:41,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:12:41,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:12:44,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:46,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:12:50,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:55,802 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:12:55,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:12:56,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:12:56,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:12:56,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:01,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:06,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:06,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:11,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:13:11,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:13:11,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:16,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:17,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:22,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:24,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:13:24,794 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:13:24,795 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:13:24,796 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:13:25,501 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:13:25,804 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:13:26,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:13:26,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:13:26,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:26,501 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:13:27,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:33,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:36,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:39,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:41,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:13:41,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:13:44,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:46,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:13:50,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:55,808 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:13:55,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:13:56,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:13:56,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:13:56,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:01,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:06,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:07,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:11,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:14:11,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:14:12,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:16,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:18,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:24,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:25,809 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:14:26,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:14:26,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:14:26,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:29,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:35,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:36,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:40,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:41,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:14:41,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:14:45,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:46,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:14:51,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:55,812 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:14:56,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:14:56,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:14:56,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:14:56,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:02,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:06,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:08,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:11,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:15:11,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:15:13,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:16,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:19,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:25,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:25,814 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:15:26,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:15:26,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:15:26,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:27,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:15:27,684 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:15:27,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:15:27,685 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:15:28,548 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:15:28,548 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:15:30,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:36,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:36,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:41,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:15:41,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:15:41,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:46,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:47,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:53,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:15:55,817 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:15:56,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:15:56,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:15:56,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:15:58,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:04,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:06,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:10,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:11,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:16:11,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:16:15,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:16,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:21,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:25,819 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:16:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:16:26,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:16:26,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:26,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:32,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:36,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:38,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:41,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:16:41,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:16:43,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:46,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:49,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:54,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:16:55,825 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:16:56,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:16:56,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:16:56,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:16:59,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:05,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:06,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:17:11,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:17:11,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:17:11,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:16,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:17:17,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:22,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:25,824 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:17:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:17:26,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:17:26,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:17:27,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:29,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:17:29,153 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:17:29,153 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:17:29,155 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:17:29,601 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:17:30,602 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:17:33,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:36,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:17:39,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:41,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:17:41,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:17:44,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:46,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:17:50,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:55,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:17:55,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:17:56,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:17:56,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:17:56,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:01,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:06,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:06,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:11,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:18:11,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:18:11,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:16,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:17,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:22,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:25,831 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:18:26,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:18:26,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:18:26,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:27,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:33,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:36,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:39,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:41,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:18:41,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:18:44,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:46,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:18:50,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:55,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:18:55,831 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:18:56,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:18:56,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:18:56,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:00,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:06,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:06,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:11,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:19:11,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:19:11,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:16,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:17,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:23,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:25,834 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:19:26,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:19:26,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:19:26,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:28,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:30,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:19:30,811 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:19:30,811 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:19:30,812 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:19:31,664 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:19:32,665 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:19:34,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:36,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:40,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:41,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:19:41,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:19:45,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:46,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:19:51,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:55,837 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:19:56,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:19:56,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:19:56,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:19:56,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:02,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:06,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:08,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:11,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:20:11,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:20:13,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:16,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:19,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:25,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:25,839 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:20:26,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:20:26,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:20:26,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:30,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:36,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:36,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:41,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:20:41,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:20:41,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:46,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:47,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:52,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:20:55,841 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:20:56,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:20:56,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:20:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:20:57,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:03,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:06,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:09,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:11,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:21:11,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:21:14,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:16,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:20,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:25,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:25,844 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:21:26,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:21:26,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:21:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:30,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:33,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:21:33,839 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:21:33,839 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:21:33,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:21:34,717 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:21:35,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:36,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:36,718 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:21:41,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:21:41,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:21:41,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:46,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:47,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:52,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:21:55,851 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:21:56,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:21:56,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:21:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:21:57,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:03,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:06,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:09,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:11,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:22:11,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:22:14,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:16,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:20,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:25,848 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:22:25,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:26,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:22:26,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:22:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:31,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:36,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:36,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:41,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:22:41,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:22:41,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:46,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:47,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:53,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:22:55,851 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:22:56,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:22:56,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:22:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:22:58,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:04,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:06,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:23:10,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:11,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:23:11,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:23:15,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:16,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:23:21,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:25,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:23:26,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:23:26,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:23:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:23:26,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:32,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:35,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:23:35,411 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:23:35,411 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:23:35,412 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:23:35,755 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:23:36,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:23:36,756 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:23:38,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:41,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:23:41,348 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:23:43,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:46,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:23:49,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:55,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:23:55,854 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:23:56,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:23:56,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:23:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:00,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:06,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:06,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:11,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:24:11,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:24:11,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:16,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:17,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:22,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:25,856 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:24:26,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:24:26,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:24:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:27,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:33,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:36,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:39,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:41,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:24:41,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:24:44,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:46,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:24:50,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:55,859 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:24:55,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:24:56,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:24:56,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:24:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:01,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:06,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:06,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:11,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:25:11,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:25:11,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:16,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:17,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:22,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:25,862 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:25:26,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:25:26,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:25:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:27,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:33,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:36,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:36,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:25:36,765 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:25:36,766 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:25:36,767 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:25:36,804 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:25:38,805 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:25:38,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:41,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:25:41,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:25:44,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:46,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:25:50,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:55,864 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:25:55,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:25:56,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:25:56,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:25:56,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:01,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:06,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:07,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:11,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:26:11,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:26:12,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:16,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:18,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:23,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:25,871 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:26:26,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:26:26,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:26:26,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:28,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:34,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:36,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:40,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:41,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:26:41,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:26:45,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:46,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:51,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:26:55,868 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:26:56,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:26:56,353 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:26:56,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:26:56,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:02,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:06,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:08,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:11,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:27:11,353 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:27:13,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:16,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:19,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:25,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:25,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:27:26,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:27:26,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:27:26,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:31,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:36,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:37,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:40,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:27:40,031 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:27:40,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:27:40,032 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:27:40,856 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:27:41,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:27:41,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:27:42,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:42,857 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:27:46,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:48,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:48,860 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:27:53,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:27:55,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:27:56,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:27:56,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:27:56,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:27:56,865 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:27:59,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:05,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:06,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:11,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:11,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:28:11,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:28:16,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:16,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:21,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:25,876 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:28:26,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:28:26,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:28:26,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:26,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:32,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:36,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:38,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:41,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:28:41,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:28:43,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:46,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:48,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:53,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:28:55,878 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:28:56,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:28:56,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:28:56,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:28:59,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:04,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:06,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:10,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:11,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:29:11,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:29:16,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:16,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:22,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:25,880 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:29:26,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:29:26,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:29:26,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:27,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:33,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:36,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:38,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:41,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:29:41,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:29:44,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:46,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:49,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:55,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:29:55,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:29:55,810 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:29:55,811 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:29:55,811 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:29:55,883 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:29:55,920 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:29:56,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:29:56,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:29:56,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:29:58,922 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:30:01,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:06,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:06,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:11,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:30:11,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:30:11,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:16,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:17,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:22,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:25,885 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:30:26,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:30:26,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:30:26,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:27,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:33,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:36,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:39,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:41,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:30:41,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:30:45,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:46,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:51,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:30:55,891 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:30:56,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:30:56,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:30:56,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:30:56,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:02,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:06,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:07,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:11,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:31:11,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:31:12,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:16,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:18,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:23,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:25,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:31:26,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:31:26,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:31:26,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:29,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:35,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:36,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:40,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:41,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:31:41,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:31:46,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:46,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:52,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:55,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:31:56,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:31:56,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:31:56,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:31:57,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:31:57,182 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:31:57,182 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:31:57,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:31:57,184 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:31:57,982 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:31:58,983 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:32:03,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:06,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:09,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:11,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:32:11,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:32:14,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:16,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:19,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:25,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:25,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:32:26,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:32:26,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:32:26,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:31,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:36,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:36,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:41,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:41,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:32:41,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:32:46,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:46,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:52,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:32:55,897 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:32:56,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:32:56,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:32:56,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:32:57,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:03,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:06,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:09,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:11,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:33:11,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:33:15,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:16,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:21,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:25,899 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:33:26,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:33:26,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:33:26,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:32,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:36,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:37,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:41,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:33:41,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:33:42,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:46,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:48,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:53,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:55,901 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:33:56,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:33:56,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:33:56,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:33:58,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:33:58,750 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:33:58,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:33:58,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:33:58,752 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:33:59,040 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:34:01,040 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:34:04,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:07,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:10,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:11,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:34:11,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:34:15,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:17,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:21,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:25,904 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:34:26,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:34:26,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:34:26,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:27,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:32,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:37,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:38,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:41,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:34:41,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:34:43,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:47,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:48,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:54,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:34:55,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:34:56,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:34:56,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:34:57,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:34:59,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:05,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:07,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:35:11,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:11,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:35:11,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:35:16,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:17,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:35:22,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:25,912 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:35:26,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:35:26,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:35:27,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:27,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:35:33,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:37,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:35:39,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:41,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:35:41,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:35:44,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:47,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:35:50,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:55,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:35:55,910 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:35:56,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:35:56,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:35:57,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:01,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:01,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:36:01,812 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:36:01,813 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:36:01,815 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:36:02,097 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:36:04,098 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:36:06,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:07,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:11,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:11,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:36:11,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:36:17,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:17,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:23,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:25,913 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:36:26,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:36:26,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:36:27,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:28,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:34,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:37,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:40,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:41,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:36:41,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:36:45,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:47,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:36:50,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:55,917 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:36:55,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:36:56,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:36:56,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:36:57,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:01,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:06,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:07,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:11,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:37:11,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:37:11,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:17,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:17,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:22,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:25,919 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:37:26,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:37:26,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:37:27,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:27,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:33,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:37,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:39,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:41,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:37:41,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:37:44,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:47,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:37:49,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:54,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:37:55,921 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:37:56,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:37:56,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:37:57,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:00,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:03,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:38:03,661 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:38:03,661 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:38:03,662 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:38:04,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:38:05,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:06,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:38:07,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:11,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:11,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:38:11,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:38:17,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:17,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:22,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:25,924 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:38:26,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:38:26,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:38:27,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:27,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:33,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:37,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:39,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:41,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:38:41,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:38:45,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:47,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:38:50,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:55,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:38:55,926 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:38:56,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:38:56,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:38:57,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:01,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:07,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:07,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:11,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:39:11,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:39:12,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:17,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:18,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:23,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:25,928 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:39:26,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:39:26,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:39:27,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:29,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:34,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:37,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:39,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:41,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:39:41,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:39:45,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:47,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:39:51,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:55,933 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:39:56,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:39:56,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:39:56,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:39:57,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:02,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:05,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:40:05,278 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:40:05,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:40:05,279 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:40:06,211 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:40:06,211 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:40:07,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:07,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:11,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:40:11,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:40:12,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:17,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:18,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:23,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:25,933 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:40:26,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:40:26,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:40:27,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:29,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:34,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:37,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:39,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:41,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:40:41,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:40:44,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:47,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:40:50,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:55,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:40:55,936 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:40:56,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:40:56,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:40:57,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:00,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:05,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:07,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:11,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:11,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:41:11,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:41:16,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:17,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:22,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:25,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:41:26,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:41:26,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:41:27,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:27,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:33,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:37,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:39,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:41,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:41:41,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:41:45,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:47,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:41:51,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:55,940 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:41:56,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:41:56,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:41:56,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:41:57,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:02,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:07,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:07,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:08,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:42:08,294 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:42:08,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:42:08,296 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:42:09,267 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:42:10,267 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:42:11,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:42:11,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:42:12,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:17,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:18,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:24,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:25,942 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:42:26,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:42:26,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:42:27,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:30,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:35,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:37,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:41,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:41,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:42:41,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:42:46,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:47,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:42:51,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:55,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:42:56,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:42:56,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:42:56,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:42:57,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:02,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:07,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:08,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:11,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:43:11,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:43:13,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:17,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:19,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:24,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:25,947 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:43:26,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:43:26,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:43:27,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:29,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:35,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:37,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:41,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:41,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:43:41,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:43:46,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:47,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:43:52,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:55,949 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:43:56,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:43:56,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:43:57,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:43:57,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:03,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:07,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:09,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:09,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:44:09,985 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:44:09,985 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:44:09,986 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:44:10,322 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:44:11,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:44:11,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:44:12,323 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:44:14,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:17,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:20,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:25,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:25,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:44:26,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:44:26,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:44:27,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:30,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:36,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:37,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:41,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:41,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:44:41,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:44:46,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:47,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:44:52,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:55,954 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:44:56,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:44:56,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:44:57,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:44:57,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:03,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:07,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:09,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:11,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:45:11,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:45:15,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:17,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:20,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:25,958 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:45:25,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:26,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:45:26,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:45:27,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:31,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:36,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:37,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:41,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:45:41,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:45:41,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:47,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:47,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:53,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:45:55,958 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:45:56,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:45:56,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:45:57,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:45:59,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:04,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:07,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:46:09,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:11,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:46:11,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:46:11,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:46:11,563 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:46:11,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:46:11,565 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:46:12,369 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:46:12,370 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:46:14,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:17,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:46:20,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:25,963 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:46:25,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:26,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:46:26,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:46:27,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:46:31,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:36,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:37,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:46:41,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:46:41,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:46:41,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:47,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:47,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:46:52,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:55,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:46:56,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:46:56,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:46:57,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:46:57,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:03,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:07,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:08,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:11,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:47:11,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:47:13,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:17,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:19,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:24,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:25,967 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:47:26,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:47:26,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:47:27,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:30,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:36,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:37,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:41,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:47:41,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:47:41,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:47,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:47,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:53,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:47:55,969 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:47:56,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:47:56,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:47:57,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:47:59,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:04,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:07,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:48:10,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:11,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:48:11,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:48:14,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:48:14,787 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:48:14,787 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:48:14,788 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:48:15,429 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:48:15,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:16,430 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:48:17,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:48:21,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:25,972 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:48:26,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:48:26,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:48:26,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:27,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:48:32,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:37,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:48:38,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:41,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:48:41,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:48:44,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:47,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:48:50,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:55,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:48:55,978 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:48:56,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:48:56,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:48:57,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:00,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:06,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:07,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:11,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:49:11,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:49:11,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:17,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:17,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:23,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:25,976 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:49:26,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:49:26,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:49:27,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:29,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:34,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:37,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:40,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:41,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:49:41,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:49:45,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:47,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:49:51,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:55,980 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:49:56,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:49:56,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:49:56,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:49:57,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:02,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:07,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:08,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:11,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:50:11,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:50:13,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:16,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:50:16,366 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:50:16,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:50:16,368 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:50:16,494 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:50:17,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:18,495 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:50:19,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:24,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:25,981 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:50:26,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:50:26,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:50:27,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:30,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:35,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:37,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:40,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:41,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:50:41,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:50:46,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:47,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:50:52,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:55,985 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:50:56,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:50:56,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:50:57,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:50:57,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:03,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:07,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:09,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:11,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:51:11,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:51:14,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:17,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:19,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:25,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:25,987 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:51:26,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:51:26,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:51:27,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:30,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:36,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:37,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:41,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:51:41,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:51:41,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:47,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:47,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:51:52,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:55,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:51:56,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:51:56,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:51:57,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:51:57,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:03,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:07,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:08,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:11,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:52:11,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:52:14,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:17,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:17,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:52:17,768 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:52:17,768 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:52:17,769 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:52:18,548 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:52:18,549 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:52:20,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:25,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:25,991 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:52:26,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:52:26,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:52:27,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:30,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:36,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:37,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:41,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:52:41,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:52:41,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:47,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:47,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:52:52,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:55,995 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:52:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:52:56,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:52:57,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:52:57,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:03,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:07,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:08,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:11,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:53:11,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:53:14,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:17,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:19,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:25,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:26,001 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:53:26,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:53:26,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:53:27,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:31,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:36,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:37,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:41,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:53:41,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:53:41,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:47,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:47,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:53:52,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:56,000 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:53:56,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:53:56,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:53:57,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:53:57,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:03,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:07,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:09,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:11,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:54:11,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:54:15,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:17,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:20,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:54:20,999 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:54:21,000 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:54:21,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:21,003 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:54:21,607 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:54:22,608 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:54:26,003 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:54:26,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:26,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:54:26,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:54:27,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:31,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:37,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:37,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:41,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:54:41,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:54:42,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:47,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:48,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:53,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:54:56,005 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:54:56,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:54:56,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:54:57,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:54:59,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:04,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:07,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:55:09,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:11,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:55:11,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:55:15,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:17,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:55:21,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:26,008 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:55:26,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:55:26,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:55:26,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:27,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:55:32,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:37,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:55:38,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:41,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:55:41,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:55:43,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:47,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:55:49,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:54,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:55:56,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:55:56,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:55:56,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:55:57,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:00,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:05,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:07,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:10,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:11,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:56:11,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:56:16,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:17,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:21,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:22,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:56:22,684 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:56:22,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:56:22,685 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:56:23,666 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:56:24,667 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:56:26,012 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:56:26,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:56:26,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:56:26,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:27,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:32,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:37,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:37,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:41,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:56:41,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:56:42,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:47,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:48,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:53,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:56:56,015 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:56:56,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:56:56,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:56:57,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:56:59,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:04,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:07,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:57:10,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:11,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:57:11,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:57:15,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:17,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:57:21,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:26,018 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:57:26,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:57:26,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:57:26,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:27,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:57:32,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:37,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:57:38,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:41,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:57:41,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:57:44,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:47,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:57:50,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:55,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:57:56,020 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:57:56,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:57:56,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:57:57,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:01,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:07,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:07,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:11,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:58:11,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:58:12,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:17,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:18,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:24,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:24,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-30 23:58:24,391 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-30 23:58:24,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-30 23:58:24,393 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-30 23:58:24,720 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-30 23:58:26,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:58:26,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:58:26,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:58:26,721 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-30 23:58:27,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:30,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:35,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:37,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:41,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:41,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:58:41,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:58:46,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:47,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:58:52,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:56,025 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:58:56,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:58:56,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:58:57,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:58:57,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:03,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:07,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:09,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:11,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:59:11,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:59:15,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:17,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:21,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:26,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:59:26,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:59:26,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:59:26,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:27,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:32,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:37,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:38,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:41,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:59:41,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:59:44,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:47,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:49,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:54,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-30 23:59:56,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-30 23:59:56,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-30 23:59:56,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-30 23:59:57,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-30 23:59:59,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:04,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:07,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:09,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:11,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:00:11,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:00:15,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:17,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:20,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:26,031 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:00:26,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:26,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:00:26,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:00:27,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:00:27,431 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:00:27,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:00:27,432 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:00:27,769 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:00:27,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:28,770 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:00:31,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:37,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:37,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:41,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:00:41,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:00:43,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:47,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:48,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:54,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:00:56,034 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:00:56,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:00:56,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:00:57,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:00:59,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:04,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:07,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:09,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:11,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:01:11,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:01:15,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:17,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:21,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:26,037 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:01:26,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:01:26,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:01:27,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:27,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:32,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:37,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:37,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:41,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:01:41,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:01:43,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:47,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:48,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:53,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:01:56,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:01:56,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:01:56,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:01:57,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:01:59,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:04,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:07,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:10,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:11,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:02:11,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:02:15,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:17,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:21,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:26,042 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:02:26,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:02:26,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:02:27,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:27,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:29,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:02:29,035 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:02:29,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:02:29,036 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:02:29,822 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:02:30,823 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:02:32,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:37,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:37,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:41,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:02:41,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:02:43,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:47,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:48,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:53,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:02:56,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:02:56,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:02:56,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:02:57,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:02:59,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:04,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:07,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:03:09,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:11,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:03:11,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:03:15,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:17,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:03:21,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:26,046 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:03:26,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:03:26,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:03:27,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:27,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:03:32,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:37,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:03:38,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:41,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:03:41,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:03:44,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:47,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:03:49,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:54,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:03:56,050 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:03:56,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:03:56,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:03:57,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:00,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:06,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:07,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:11,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:04:11,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:04:12,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:17,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:17,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:22,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:26,051 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:04:26,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:04:26,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:04:27,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:28,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:30,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:04:30,571 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:04:30,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:04:30,573 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:04:30,886 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:04:32,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:04:33,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:37,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:38,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:41,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:04:41,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:04:44,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:47,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:04:49,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:55,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:04:56,054 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:04:56,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:04:56,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:04:57,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:00,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:06,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:07,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:11,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:05:11,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:05:12,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:17,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:17,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:23,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:26,056 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:05:26,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:05:26,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:05:27,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:29,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:34,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:37,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:40,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:41,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:05:41,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:05:46,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:47,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:05:51,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:56,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:05:56,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:05:56,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:05:57,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:05:57,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:02,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:07,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:08,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:11,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:06:11,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:06:14,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:17,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:19,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:25,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:26,061 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:06:26,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:06:26,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:06:27,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:31,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:33,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:06:33,775 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:06:33,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:06:33,776 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:06:33,936 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:06:34,937 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:06:37,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:37,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:41,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:06:41,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:06:43,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:47,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:06:48,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:54,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:06:56,063 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:06:56,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:06:56,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:06:57,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:00,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:06,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:07,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:11,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:07:11,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:07:12,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:17,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:17,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:22,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:26,071 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:07:26,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:07:26,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:07:27,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:28,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:33,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:37,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:38,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:41,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:07:41,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:07:44,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:47,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:07:49,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:55,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:07:56,068 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:07:56,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:07:56,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:07:57,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:00,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:06,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:07,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:11,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:08:11,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:08:12,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:17,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:17,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:22,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:26,072 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:08:26,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:08:26,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:08:27,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:28,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:33,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:35,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:08:35,247 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:08:35,247 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:08:35,248 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:08:35,987 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:08:36,987 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:08:37,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:39,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:41,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:08:41,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:08:44,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:47,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:08:49,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:55,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:08:56,073 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:08:56,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:08:56,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:08:57,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:00,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:06,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:07,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:11,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:09:11,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:09:12,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:17,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:17,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:22,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:26,075 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:09:26,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:09:26,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:09:27,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:28,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:33,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:37,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:38,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:41,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:09:41,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:09:44,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:47,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:09:49,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:55,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:09:56,077 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:09:56,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:09:56,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:09:57,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:00,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:05,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:07,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:11,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:10:11,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:10:11,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:16,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:17,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:22,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:26,080 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:10:26,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:10:26,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:10:27,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:28,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:33,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:36,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:10:36,622 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:10:36,622 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:10:36,623 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:10:37,044 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:10:37,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:39,045 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:10:39,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:41,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:10:41,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:10:44,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:47,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:10:49,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:54,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:10:56,082 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:10:56,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:10:56,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:10:57,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:00,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:06,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:07,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:11,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:11:11,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:11:12,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:17,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:17,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:23,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:26,085 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:11:26,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:11:26,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:11:27,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:29,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:34,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:37,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:40,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:41,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:11:41,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:11:45,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:47,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:11:51,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:56,092 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:11:56,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:11:56,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:11:57,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:11:57,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:02,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:07,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:07,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:11,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:12:11,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:12:13,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:17,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:18,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:24,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:26,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:12:26,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:12:26,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:12:27,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:29,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:34,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:37,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:38,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:12:38,051 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:12:38,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:12:38,054 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:12:38,100 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:12:39,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:12:40,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:41,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:12:41,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:12:45,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:47,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:12:50,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:55,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:12:56,092 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:12:56,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:12:56,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:12:57,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:01,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:07,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:07,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:11,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:13:11,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:13:13,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:17,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:18,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:24,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:26,096 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:13:26,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:13:26,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:13:27,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:30,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:36,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:37,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:41,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:13:41,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:13:42,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:47,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:47,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:53,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:13:56,097 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:13:56,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:13:56,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:13:57,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:13:59,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:04,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:07,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:10,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:11,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:14:11,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:14:15,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:17,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:21,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:26,099 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:14:26,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:14:26,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:14:27,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:27,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:32,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:37,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:38,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:41,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:14:41,200 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:14:41,200 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:14:41,201 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:14:41,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:14:41,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:14:42,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:14:43,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:14:44,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:47,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:49,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:54,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:14:56,102 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:14:56,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:14:56,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:14:57,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:14:59,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:05,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:07,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:15:11,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:15:11,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:15:11,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:16,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:17,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:15:22,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:26,104 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:15:26,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:15:26,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:15:27,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:15:28,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:33,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:37,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:15:39,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:41,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:15:41,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:15:45,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:47,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:15:51,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:56,107 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:15:56,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:15:56,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:15:57,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:15:57,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:02,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:07,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:07,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:11,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:16:11,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:16:13,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:17,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:18,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:23,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:26,109 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:16:26,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:16:26,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:16:27,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:29,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:34,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:37,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:40,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:41,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:16:41,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:16:42,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:16:42,787 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:16:42,787 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:16:42,788 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:16:43,214 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:16:45,215 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:16:46,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:47,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:52,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:16:56,111 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:16:56,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:16:56,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:16:57,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:16:58,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:03,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:07,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:08,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:11,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:17:11,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:17:14,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:17,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:19,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:24,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:26,113 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:17:26,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:17:26,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:17:27,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:30,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:36,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:37,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:41,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:17:41,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:17:42,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:47,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:47,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:52,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:17:56,116 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:17:56,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:17:56,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:17:57,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:17:58,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:03,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:07,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:09,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:11,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:18:11,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:18:15,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:17,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:20,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:26,118 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:18:26,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:26,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:18:26,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:18:27,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:31,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:36,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:37,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:41,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:18:41,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:18:42,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:44,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:18:44,362 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:18:44,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:18:44,363 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:18:45,265 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:18:45,265 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:18:47,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:47,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:52,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:18:56,120 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:18:56,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:18:56,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:18:57,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:18:58,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:03,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:07,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:19:08,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:11,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:19:11,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:19:14,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:17,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:19:19,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:24,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:26,122 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:19:26,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:19:26,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:19:27,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:19:30,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:35,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:37,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:19:41,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:41,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:19:41,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:19:46,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:47,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:19:51,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:56,125 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:19:56,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:19:56,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:19:56,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:19:57,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:02,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:07,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:08,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:11,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:20:11,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:20:13,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:17,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:19,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:24,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:26,128 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:20:26,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:20:26,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:20:27,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:29,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:35,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:37,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:41,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:41,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:20:41,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:20:47,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:47,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:20:47,388 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:20:47,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:20:47,389 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:20:47,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:20:48,318 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:20:49,319 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:20:52,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:56,129 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:20:56,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:20:56,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:20:57,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:20:57,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:03,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:07,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:08,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:11,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:21:11,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:21:13,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:17,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:19,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:24,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:26,135 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:21:26,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:21:26,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:21:27,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:29,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:35,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:37,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:41,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:41,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:21:41,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:21:47,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:47,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:21:52,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:56,134 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:21:56,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:21:56,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:21:57,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:21:57,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:03,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:07,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:09,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:11,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:22:11,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:22:14,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:17,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:20,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:25,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:26,137 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:22:26,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:22:26,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:22:27,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:30,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:36,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:37,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:41,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:22:41,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:22:41,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:47,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:47,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:48,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:22:48,828 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:22:48,828 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:22:48,829 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:22:49,378 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:22:51,379 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:22:53,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:22:56,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:22:56,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:22:56,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:22:57,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:22:58,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:04,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:07,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:10,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:11,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:23:11,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:23:15,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:17,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:20,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:26,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:26,141 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:23:26,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:23:26,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:23:27,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:31,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:37,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:37,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:41,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:23:41,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:23:42,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:47,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:48,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:54,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:23:56,143 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:23:56,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:23:56,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:23:57,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:23:59,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:05,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:07,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:24:10,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:11,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:24:11,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:24:16,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:17,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:24:22,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:26,146 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:24:26,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:24:26,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:24:27,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:27,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:24:33,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:37,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:24:39,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:41,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:24:41,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:24:44,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:47,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:24:50,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:50,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:24:50,395 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:24:50,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:24:50,396 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:24:50,437 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:24:51,438 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:24:56,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:24:56,148 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:24:56,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:24:56,441 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:24:57,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:01,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:07,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:07,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:11,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:25:11,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:25:12,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:17,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:18,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:24,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:26,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:25:26,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:25:26,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:25:27,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:29,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:35,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:37,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:40,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:41,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:25:41,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:25:45,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:47,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:25:50,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:55,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:25:56,156 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:25:56,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:25:56,441 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:25:57,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:01,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:06,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:07,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:11,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:26:11,441 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:26:11,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:17,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:17,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:23,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:26,154 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:26:26,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:26:26,441 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:26:27,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:28,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:34,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:37,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:39,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:41,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:26:41,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:26:44,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:47,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:26:50,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:53,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:26:53,507 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:26:53,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:26:53,509 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:26:54,500 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:26:55,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:26:55,500 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:26:56,158 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:26:56,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:26:56,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:26:57,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:01,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:07,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:07,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:11,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:27:11,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:27:12,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:17,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:18,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:24,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:26,161 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:27:26,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:27:26,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:27:27,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:29,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:35,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:37,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:40,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:41,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:27:41,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:27:46,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:47,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:27:51,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:56,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:27:56,162 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:27:56,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:27:56,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:27:57,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:02,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:07,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:08,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:11,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:28:11,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:28:13,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:17,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:19,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:25,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:26,165 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:28:26,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:28:26,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:28:27,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:31,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:36,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:37,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:41,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:28:41,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:28:41,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:47,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:47,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:28:52,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:54,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:28:54,973 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:28:54,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:28:54,974 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:28:55,545 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:28:56,167 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:28:56,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:28:56,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:28:57,546 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:28:57,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:28:57,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:03,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:07,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:09,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:11,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:29:11,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:29:14,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:17,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:20,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:25,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:26,170 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:29:26,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:29:26,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:29:27,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:31,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:36,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:37,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:41,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:29:41,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:29:41,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:47,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:47,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:53,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:29:56,172 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:29:56,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:29:56,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:29:57,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:29:58,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:04,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:07,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:10,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:11,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:30:11,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:30:15,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:17,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:21,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:26,178 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:30:26,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:26,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:30:26,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:30:27,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:32,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:37,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:37,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:41,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:30:41,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:30:42,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:47,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:48,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:53,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:56,176 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:30:56,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:30:56,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:30:56,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:30:56,658 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:30:56,660 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:30:56,662 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:30:57,606 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:30:57,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:30:58,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:30:59,607 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:31:04,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:07,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:09,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:11,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:31:11,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:31:14,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:17,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:20,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:26,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:26,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:31:26,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:31:26,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:31:27,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:31,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:37,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:37,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:41,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:31:41,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:31:42,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:47,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:48,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:54,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:31:56,183 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:31:56,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:31:56,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:31:57,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:31:59,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:05,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:07,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:11,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:11,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:32:11,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:32:16,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:17,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:22,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:26,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:32:26,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:32:26,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:32:27,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:27,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:33,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:37,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:39,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:41,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:32:41,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:32:44,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:47,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:50,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:56,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:32:56,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:32:56,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:32:56,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:32:57,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:32:59,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:32:59,747 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:32:59,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:32:59,748 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:33:00,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:33:01,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:01,657 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:33:06,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:07,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:33:11,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:33:11,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:33:11,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:17,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:17,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:33:23,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:26,189 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:33:26,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:33:26,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:33:27,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:33:28,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:34,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:37,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:33:39,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:41,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:33:41,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:33:44,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:47,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:33:50,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:56,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:33:56,191 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:33:56,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:33:56,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:33:57,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:01,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:07,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:07,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:11,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:34:11,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:34:12,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:17,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:18,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:24,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:26,194 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:34:26,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:34:26,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:34:27,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:29,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:35,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:37,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:41,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:41,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:34:41,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:34:47,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:47,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:34:52,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:56,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:34:56,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:34:56,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:34:57,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:34:57,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:01,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:35:01,170 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:35:01,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:35:01,171 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:35:01,704 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:35:03,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:03,705 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:35:07,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:09,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:11,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:35:11,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:35:14,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:17,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:20,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:26,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:26,199 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:35:26,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:35:26,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:35:27,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:31,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:37,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:37,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:41,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:35:41,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:35:42,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:47,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:48,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:54,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:35:56,202 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:35:56,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:35:56,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:35:57,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:35:59,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:05,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:07,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:36:11,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:11,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:36:11,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:36:16,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:17,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:36:22,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:26,204 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:36:26,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:36:26,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:36:27,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:27,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:36:33,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:37,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:36:39,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:41,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:36:41,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:36:44,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:47,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:36:50,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:55,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:36:56,207 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:36:56,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:36:56,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:36:57,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:01,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:02,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:37:02,775 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:37:02,776 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:37:02,776 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:37:03,750 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:37:05,751 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:37:06,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:07,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:11,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:11,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:37:11,466 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:37:17,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:17,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:23,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:26,209 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:37:26,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:37:26,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:37:27,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:28,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:34,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:37,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:40,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:41,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:37:41,466 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:37:45,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:47,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:37:50,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:56,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:37:56,211 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:37:56,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:37:56,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:37:57,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:02,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:07,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:08,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:11,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:38:11,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:38:13,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:17,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:19,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:24,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:26,213 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:38:26,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:38:26,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:38:27,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:29,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:35,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:37,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:40,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:41,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:38:41,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:38:45,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:47,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:38:51,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:56,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:38:56,215 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:38:56,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:38:56,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:38:57,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:02,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:05,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:39:05,733 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:39:05,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:39:05,734 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:39:05,800 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:39:07,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:07,801 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:39:07,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:11,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:39:11,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:39:13,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:17,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:19,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:25,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:26,283 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:39:26,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:39:26,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:39:27,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:30,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:36,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:37,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:41,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:39:41,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:39:41,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:47,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:47,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:53,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:39:56,289 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:39:56,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:39:56,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:39:57,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:39:58,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:04,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:07,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:10,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:11,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:40:11,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:40:15,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:17,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:21,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:26,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:26,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:40:26,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:40:26,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:40:27,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:32,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:37,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:37,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:41,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:40:41,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:40:42,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:47,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:48,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:53,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:40:56,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:40:56,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:40:56,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:40:57,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:40:58,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:04,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:07,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:41:07,179 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:41:07,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:41:07,180 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:41:07,857 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:41:07,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:09,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:09,858 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:41:11,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:41:11,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:41:14,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:17,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:20,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:25,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:26,293 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:41:26,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:41:26,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:41:27,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:31,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:36,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:37,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:41,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:41:41,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:41:41,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:47,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:47,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:53,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:41:56,295 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:41:56,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:41:56,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:41:57,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:41:58,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:04,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:07,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:10,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:11,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:42:11,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:42:15,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:17,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:21,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:26,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:26,297 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:42:26,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:42:26,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:42:27,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:32,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:37,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:37,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:41,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:42:41,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:42:42,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:47,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:48,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:54,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:42:56,300 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:42:56,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:42:56,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:42:57,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:42:59,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:05,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:07,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:08,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:43:08,628 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:43:08,628 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:43:08,629 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:43:08,907 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:43:09,908 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:43:10,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:11,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:43:11,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:43:15,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:17,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:21,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:26,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:26,302 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:43:26,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:43:26,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:43:27,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:32,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:37,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:37,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:41,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:43:41,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:43:42,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:47,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:48,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:54,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:43:56,305 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:43:56,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:43:56,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:43:57,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:43:59,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:05,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:07,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:44:11,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:11,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:44:11,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:44:17,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:17,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:44:23,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:26,311 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:44:26,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:44:26,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:44:27,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:44:28,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:34,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:37,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:44:39,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:41,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:44:41,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:44:44,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:47,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:44:50,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:56,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:44:56,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:44:56,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:44:56,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:44:57,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:01,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:07,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:07,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:10,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:45:10,054 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:45:10,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:45:10,056 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:45:10,954 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:45:11,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:45:11,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:45:11,955 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:45:12,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:18,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:19,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:23,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:26,312 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:45:26,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:45:26,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:45:28,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:29,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:34,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:39,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:40,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:41,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:45:41,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:45:46,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:49,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:45:51,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:56,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:45:56,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:45:56,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:45:56,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:45:59,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:02,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:07,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:09,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:11,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:46:11,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:46:12,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:18,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:19,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:24,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:26,317 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:46:26,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:46:26,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:46:29,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:29,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:35,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:39,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:41,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:41,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:46:41,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:46:46,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:49,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:46:51,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:56,321 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:46:56,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:46:56,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:46:56,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:46:59,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:02,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:07,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:09,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:11,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:47:11,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:47:12,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:13,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:47:13,036 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:47:13,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:47:13,038 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:47:14,007 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:47:16,008 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:47:18,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:19,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:23,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:26,323 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:47:26,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:47:26,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:47:28,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:29,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:34,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:39,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:40,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:41,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:47:41,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:47:45,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:49,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:47:50,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:56,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:47:56,325 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:47:56,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:47:56,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:47:59,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:02,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:07,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:09,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:11,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:48:11,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:48:12,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:18,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:19,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:24,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:26,327 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:48:26,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:48:26,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:48:29,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:29,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:35,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:39,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:41,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:41,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:48:41,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:48:47,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:49,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:48:53,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:56,335 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:48:56,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:48:56,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:48:58,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:48:59,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:04,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:09,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:10,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:11,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:49:11,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:49:14,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:49:14,507 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:49:14,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:49:14,508 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:49:15,058 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:49:15,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:16,059 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:49:19,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:21,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:26,332 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:49:26,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:26,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:49:26,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:49:29,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:32,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:37,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:39,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:41,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:49:41,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:49:42,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:48,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:49,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:49:53,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:56,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:49:56,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:49:56,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:49:58,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:49:59,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:04,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:09,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:10,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:11,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:50:11,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:50:15,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:19,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:20,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:26,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:26,337 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:50:26,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:50:26,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:50:29,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:32,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:37,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:39,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:41,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:50:41,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:50:42,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:48,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:49,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:50:53,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:56,340 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:50:56,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:50:56,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:50:58,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:50:59,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:04,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:09,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:10,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:11,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:51:11,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:51:15,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:15,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:51:15,992 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:51:15,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:51:15,993 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:51:16,110 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:51:17,111 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:51:19,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:21,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:26,342 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:51:26,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:26,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:51:26,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:51:29,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:32,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:38,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:39,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:41,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:51:41,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:51:43,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:49,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:51:49,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:55,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:51:56,345 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:51:56,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:51:56,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:51:59,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:00,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:06,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:09,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:11,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:52:11,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:52:11,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:17,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:19,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:23,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:26,347 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:52:26,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:52:26,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:52:28,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:29,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:34,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:39,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:39,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:41,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:52:41,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:52:44,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:49,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:52:50,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:56,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:52:56,349 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:52:56,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:52:56,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:52:59,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:01,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:07,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:09,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:11,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:53:11,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:53:12,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:18,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:19,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:19,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:53:19,217 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:53:19,218 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:53:19,218 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:53:20,162 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:53:21,162 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:53:23,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:26,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:53:26,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:53:26,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:53:28,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:29,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:34,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:39,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:40,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:41,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:53:41,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:53:45,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:49,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:53:51,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:56,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:53:56,358 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:53:56,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:53:56,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:53:59,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:02,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:07,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:09,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:11,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:54:11,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:54:12,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:18,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:19,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:24,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:26,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:54:26,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:54:26,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:54:29,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:29,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:35,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:39,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:40,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:41,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:54:41,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:54:45,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:49,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:54:51,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:56,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:54:56,360 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:54:56,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:54:56,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:54:59,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:02,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:07,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:09,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:11,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:55:11,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:55:12,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:18,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:19,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:20,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:55:20,727 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:55:20,728 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:55:20,728 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:55:21,216 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:55:21,217 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:55:24,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:26,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:55:26,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:55:26,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:55:29,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:29,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:35,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:39,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:41,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:41,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:55:41,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:55:47,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:49,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:55:52,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:56,364 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:55:56,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:55:56,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:55:57,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:55:59,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:03,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:09,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:09,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:11,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:56:11,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:56:14,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:19,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:20,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:26,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:26,366 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:56:26,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:56:26,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:56:29,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:32,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:37,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:39,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:41,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:56:41,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:56:42,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:48,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:49,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:54,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:56:56,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:56:56,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:56:56,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:56:59,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:56:59,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:05,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:09,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:11,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:11,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:57:11,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:57:16,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:19,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:22,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:22,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:57:22,304 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:57:22,304 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:57:22,305 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:57:23,261 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:57:23,261 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:57:26,371 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:57:26,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:57:26,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:57:27,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:29,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:33,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:38,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:39,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:41,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:57:41,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:57:43,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:49,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:49,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:54,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:57:56,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:57:56,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:57:56,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:57:59,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:57:59,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:05,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:09,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:58:11,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:11,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:58:11,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:58:16,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:19,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:58:21,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:26,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:26,379 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:58:26,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:58:26,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:58:29,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:58:32,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:37,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:39,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:58:41,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:58:41,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:58:42,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:48,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:49,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:58:53,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:56,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:58:56,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:58:56,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:58:58,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:58:59,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:04,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:09,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:10,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:11,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:59:11,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:59:15,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:19,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:21,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:25,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 00:59:25,173 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 00:59:25,173 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 00:59:25,175 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 00:59:25,318 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 00:59:26,381 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:59:26,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:26,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:59:26,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:59:27,319 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 00:59:29,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:32,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:37,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:39,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:41,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:59:41,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:59:42,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:48,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:49,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:54,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 00:59:56,383 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 00:59:56,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 00:59:56,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 00:59:59,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 00:59:59,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:05,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:09,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:11,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:11,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:00:11,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:00:16,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:19,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:21,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:26,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:26,386 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:00:26,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:00:26,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:00:29,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:32,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:37,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:39,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:41,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:00:41,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:00:42,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:48,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:49,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:54,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:00:56,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:00:56,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:00:56,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:00:59,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:00:59,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:05,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:09,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:01:11,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:11,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:01:11,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:01:17,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:19,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:01:23,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:26,391 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:01:26,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:01:26,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:01:26,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:01:26,622 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:01:26,623 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:01:26,624 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:01:27,381 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:01:27,381 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:01:28,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:29,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:01:34,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:39,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:01:39,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:41,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:01:41,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:01:44,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:49,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:01:50,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:56,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:01:56,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:01:56,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:01:56,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:01:59,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:02,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:07,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:09,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:11,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:02:11,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:02:12,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:18,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:19,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:23,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:26,396 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:02:26,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:02:26,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:02:28,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:29,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:34,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:39,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:39,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:41,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:02:41,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:02:44,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:49,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:02:50,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:56,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:02:56,396 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:02:56,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:02:56,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:02:59,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:01,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:07,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:09,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:11,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:03:11,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:03:12,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:18,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:19,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:24,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:26,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:03:26,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:03:26,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:03:28,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:03:28,130 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:03:28,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:03:28,131 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:03:28,440 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:03:29,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:29,441 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:03:30,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:35,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:39,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:41,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:41,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:03:41,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:03:47,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:49,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:03:53,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:56,401 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:03:56,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:03:56,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:03:58,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:03:59,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:04,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:09,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:09,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:11,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:04:11,502 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:04:14,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:19,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:20,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:25,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:26,406 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:04:26,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:04:26,502 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:04:29,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:30,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:36,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:39,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:41,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:04:41,502 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:04:41,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:47,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:49,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:04:53,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:56,406 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:04:56,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:04:56,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:04:58,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:04:59,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:04,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:09,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:10,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:11,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:05:11,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:05:15,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:19,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:21,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:26,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:26,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:05:26,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:05:26,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:05:29,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:31,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:31,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:05:31,350 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:05:31,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:05:31,351 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:05:31,508 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:05:33,509 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:05:36,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:39,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:41,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:41,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:05:41,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:05:47,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:49,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:05:52,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:56,411 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:05:56,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:05:56,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:05:57,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:05:59,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:03,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:08,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:09,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:11,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:06:11,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:06:13,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:19,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:19,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:24,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:26,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:06:26,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:06:26,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:06:29,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:29,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:35,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:39,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:41,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:41,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:06:41,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:06:47,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:49,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:06:52,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:56,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:06:56,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:06:56,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:06:57,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:06:59,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:03,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:09,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:09,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:11,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:07:11,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:07:14,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:19,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:20,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:26,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:26,418 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:07:26,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:07:26,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:07:29,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:31,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:32,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:07:32,794 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:07:32,794 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:07:32,795 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:07:33,569 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:07:33,570 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:07:36,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:39,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:41,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:07:41,506 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:07:41,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:47,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:49,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:07:53,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:56,424 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:07:56,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:07:56,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:07:58,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:07:59,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:04,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:09,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:10,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:11,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:08:11,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:08:15,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:19,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:21,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:26,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:08:26,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:26,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:08:26,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:08:29,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:32,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:37,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:39,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:41,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:08:41,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:08:42,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:48,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:49,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:08:53,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:56,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:08:56,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:08:56,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:08:58,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:08:59,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:04,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:09,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:10,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:11,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:09:11,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:09:15,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:19,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:21,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:26,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:26,427 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:09:26,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:09:26,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:09:29,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:32,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:34,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:09:34,324 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:09:34,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:09:34,325 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:09:34,617 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:09:35,617 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:09:37,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:39,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:41,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:09:41,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:09:42,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:48,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:49,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:09:53,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:56,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:09:56,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:09:56,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:09:58,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:09:59,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:04,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:09,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:10,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:11,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:10:11,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:10:15,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:19,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:21,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:26,432 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:10:26,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:26,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:10:26,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:10:29,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:32,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:37,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:39,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:41,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:10:41,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:10:42,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:48,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:49,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:54,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:10:56,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:10:56,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:10:56,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:10:59,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:10:59,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:05,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:09,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:11:10,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:11,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:11:11,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:11:16,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:19,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:11:21,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:26,436 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:11:26,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:11:26,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:11:27,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:29,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:11:32,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:37,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:11:37,462 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:11:37,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:11:37,463 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:11:37,677 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:11:38,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:39,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:11:39,678 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:11:41,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:11:41,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:11:44,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:49,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:11:49,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:55,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:11:56,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:11:56,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:11:56,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:11:59,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:00,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:05,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:09,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:11,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:12:11,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:12:11,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:16,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:19,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:22,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:26,446 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:12:26,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:12:26,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:12:27,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:29,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:32,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:38,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:39,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:41,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:12:41,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:12:44,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:49,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:12:49,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:55,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:12:56,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:12:56,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:12:56,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:12:59,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:00,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:06,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:09,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:11,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:13:11,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:13:12,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:17,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:19,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:22,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:26,447 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:13:26,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:13:26,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:13:28,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:29,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:33,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:38,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:13:38,851 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:13:38,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:13:38,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:38,852 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:13:39,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:39,730 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:13:39,730 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:13:41,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:13:41,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:13:44,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:49,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:13:50,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:56,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:13:56,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:13:56,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:13:56,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:13:59,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:01,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:07,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:09,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:11,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:14:11,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:14:13,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:18,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:19,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:23,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:26,450 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:14:26,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:14:26,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:14:29,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:29,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:34,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:39,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:39,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:41,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:14:41,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:14:45,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:49,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:14:51,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:56,453 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:14:56,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:14:56,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:14:57,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:14:59,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:02,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:08,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:09,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:11,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:15:11,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:15:14,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:19,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:19,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:24,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:26,455 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:15:26,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:15:26,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:15:29,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:29,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:35,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:39,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:40,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:15:40,201 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:15:40,201 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:15:40,202 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:15:40,776 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:15:41,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:41,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:15:41,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:15:41,776 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:15:46,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:49,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:15:51,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:56,458 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:15:56,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:15:56,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:15:57,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:15:59,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:02,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:07,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:09,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:11,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:16:11,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:16:13,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:18,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:19,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:24,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:26,460 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:16:26,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:16:26,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:16:29,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:29,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:34,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:39,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:40,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:41,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:16:41,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:16:46,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:49,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:16:52,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:56,463 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:16:56,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:16:56,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:16:58,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:16:59,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:03,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:09,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:09,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:11,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:17:11,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:17:14,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:19,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:20,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:26,470 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:17:26,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:26,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:17:26,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:17:29,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:31,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:37,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:39,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:41,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:17:41,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:17:43,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:17:43,612 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:17:43,613 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:17:43,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:43,614 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:17:43,835 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:17:45,835 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:17:48,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:49,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:17:54,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:17:56,467 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:17:56,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:17:56,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:17:59,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:00,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:05,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:09,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:11,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:18:11,538 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:18:11,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:16,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:19,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:21,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:26,471 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:18:26,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:18:26,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:18:27,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:29,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:32,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:37,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:39,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:41,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:18:41,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:18:43,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:49,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:18:49,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:55,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:18:56,472 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:18:56,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:18:56,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:18:59,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:01,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:06,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:09,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:11,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:19:11,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:19:12,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:17,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:19,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:22,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:26,475 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:19:26,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:19:26,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:19:27,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:29,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:32,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:37,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:39,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:41,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:19:41,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:19:43,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:44,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:19:44,941 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:19:44,941 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:19:44,942 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:19:45,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:19:45,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:19:48,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:49,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:53,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:19:56,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:19:56,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:19:56,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:19:59,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:19:59,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:05,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:09,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:11,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:20:11,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:20:11,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:16,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:19,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:21,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:26,479 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:20:26,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:20:26,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:20:27,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:29,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:32,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:37,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:39,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:41,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:20:41,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:20:43,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:48,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:49,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:54,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:20:56,482 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:20:56,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:20:56,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:20:59,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:20:59,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:05,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:09,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:21:10,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:11,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:21:11,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:21:15,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:19,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:21:20,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:25,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:26,484 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:21:26,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:21:26,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:21:29,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:21:30,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:36,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:39,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:21:41,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:21:41,546 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:21:42,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:46,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:21:46,568 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:21:46,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:21:46,569 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:21:46,942 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:21:47,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:47,942 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:21:49,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:21:52,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:56,491 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:21:56,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:21:56,546 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:21:57,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:21:59,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:03,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:08,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:09,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:11,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:22:11,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:22:13,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:19,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:19,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:24,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:26,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:22:26,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:22:26,546 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:22:29,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:30,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:35,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:39,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:41,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:22:41,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:22:41,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:46,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:49,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:22:51,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:56,492 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:22:56,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:22:56,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:22:57,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:22:59,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:02,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:07,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:09,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:11,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:23:11,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:23:12,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:17,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:19,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:23,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:26,493 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:23:26,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:23:26,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:23:29,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:29,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:34,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:39,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:40,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:41,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:23:41,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:23:46,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:48,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:23:48,059 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:23:48,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:23:48,059 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:23:48,992 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:23:49,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:23:49,992 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:23:52,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:56,495 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:23:56,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:23:56,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:23:57,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:23:59,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:02,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:08,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:09,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:11,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:24:11,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:24:14,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:19,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:20,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:25,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:26,498 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:24:26,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:24:26,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:24:29,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:31,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:37,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:39,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:41,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:24:41,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:24:43,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:48,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:49,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:53,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:24:56,500 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:24:56,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:24:56,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:24:59,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:24:59,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:05,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:09,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:25:11,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:25:11,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:25:11,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:16,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:19,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:25:22,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:26,503 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:25:26,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:25:26,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:25:27,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:29,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:25:33,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:38,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:39,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:25:41,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:25:41,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:25:44,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:49,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:25:50,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:50,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:25:50,994 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:25:50,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:25:50,994 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:25:51,041 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:25:52,041 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:25:56,505 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:25:56,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:25:56,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:25:56,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:25:59,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:01,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:06,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:09,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:11,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:26:11,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:26:12,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:17,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:19,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:23,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:26,511 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:26:26,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:26:26,551 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:26:29,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:29,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:34,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:39,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:40,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:41,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:26:41,551 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:26:46,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:49,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:26:51,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:56,510 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:26:56,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:26:56,551 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:26:57,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:26:59,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:02,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:08,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:09,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:11,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:27:11,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:27:13,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:19,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:19,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:24,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:26,513 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:27:26,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:27:26,551 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:27:29,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:30,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:35,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:39,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:41,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:27:41,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:27:41,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:46,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:49,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:27:52,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:27:52,367 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:27:52,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:27:52,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:52,369 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:27:53,080 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:27:54,081 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:27:56,514 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:27:56,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:27:56,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:27:57,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:27:59,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:02,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:07,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:09,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:11,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:28:11,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:28:13,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:18,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:19,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:24,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:26,516 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:28:26,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:28:26,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:28:29,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:30,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:35,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:39,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:41,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:28:41,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:28:41,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:46,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:49,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:28:51,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:56,519 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:28:56,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:28:56,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:28:57,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:28:59,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:02,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:07,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:09,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:11,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:29:11,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:29:13,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:18,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:19,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:23,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:26,521 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:29:26,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:29:26,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:29:28,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:29,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:34,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:39,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:40,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:41,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:29:41,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:29:46,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:49,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:29:51,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:53,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:29:53,978 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:29:53,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:29:53,979 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:29:54,134 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:29:56,135 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:29:56,523 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:29:56,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:29:56,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:29:57,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:29:59,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:02,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:07,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:09,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:11,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:30:11,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:30:12,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:18,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:19,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:24,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:26,526 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:30:26,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:30:26,555 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:30:29,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:30,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:35,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:39,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:41,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:30:41,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:30:41,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:46,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:49,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:30:52,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:56,531 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:30:56,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:30:56,555 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:30:57,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:30:59,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:03,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:09,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:09,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:11,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:31:11,555 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:31:15,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:19,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:20,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:26,530 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:31:26,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:26,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:31:26,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:31:29,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:31,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:37,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:39,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:41,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:31:41,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:31:43,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:48,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:49,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:53,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:31:56,534 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:31:56,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:31:56,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:31:56,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:31:56,936 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:31:56,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:31:56,937 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:31:57,179 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:31:58,179 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:31:59,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:31:59,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:05,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:09,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:32:10,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:11,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:32:11,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:32:15,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:19,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:32:20,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:26,534 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:32:26,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:26,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:32:26,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:32:29,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:32:31,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:37,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:39,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:32:41,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:32:41,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:32:42,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:48,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:49,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:32:54,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:32:56,537 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:32:56,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:32:56,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:32:59,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:00,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:06,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:09,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:11,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:33:11,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:33:12,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:17,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:19,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:22,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:26,539 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:33:26,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:33:26,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:33:28,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:29,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:33,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:39,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:39,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:41,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:33:41,558 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:33:45,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:49,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:33:51,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:56,542 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:33:56,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:33:56,558 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:33:57,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:33:58,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:33:58,296 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:33:58,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:33:58,297 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:33:59,225 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:33:59,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:00,225 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:34:02,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:08,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:09,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:11,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:34:11,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:34:13,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:19,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:19,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:25,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:26,544 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:34:26,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:34:26,558 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:34:29,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:31,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:36,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:39,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:41,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:34:41,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:34:42,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:47,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:49,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:53,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:34:56,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:34:56,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:34:56,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:34:59,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:34:59,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:04,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:09,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:10,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:11,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:35:11,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:35:16,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:19,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:22,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:26,548 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:35:26,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:35:26,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:35:27,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:29,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:32,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:37,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:39,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:41,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:35:41,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:35:43,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:49,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:49,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:54,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:56,554 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:35:56,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:35:56,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:35:59,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:35:59,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:35:59,921 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:35:59,922 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:35:59,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:35:59,930 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:36:00,278 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:36:02,279 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:36:05,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:09,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:36:11,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:36:11,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:36:11,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:16,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:19,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:36:22,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:26,553 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:36:26,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:36:26,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:36:28,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:29,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:36:34,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:39,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:36:39,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:41,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:36:41,561 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:36:45,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:49,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:36:51,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:56,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:36:56,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:36:56,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:36:57,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:36:59,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:02,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:08,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:09,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:11,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:37:11,561 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:37:13,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:18,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:19,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:24,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:26,557 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:37:26,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:37:26,561 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:37:29,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:30,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:35,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:39,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:40,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:41,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:37:41,561 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:37:46,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:49,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:37:52,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:56,559 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:37:56,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:37:56,563 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:37:57,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:37:59,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:02,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:38:02,990 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:38:02,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:38:02,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:02,992 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:38:03,328 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:38:04,329 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:38:08,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:09,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:11,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:38:11,562 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:38:14,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:19,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:20,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:26,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:38:26,563 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:38:26,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:38:29,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:32,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:38,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:39,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:41,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:38:41,562 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:38:43,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:49,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:49,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:38:55,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:38:56,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:38:56,564 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:38:56,565 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:38:59,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:00,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:05,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:09,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:11,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:11,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:39:11,563 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:39:17,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:19,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:22,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:39:26,567 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:39:26,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:39:27,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:29,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:33,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:38,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:39,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:41,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:39:41,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:39:44,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:49,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:39:50,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:56,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:39:56,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:39:56,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:39:56,744 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:39:59,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:01,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:04,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:40:04,346 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:40:04,346 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:40:04,346 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:40:04,398 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:40:06,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:06,399 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:40:09,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:11,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:40:11,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:40:11,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:17,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:19,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:23,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:26,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:40:26,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:40:26,700 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:40:28,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:29,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:34,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:39,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:40,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:41,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:40:41,565 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:40:45,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:49,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:40:51,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:56,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:40:56,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:40:56,565 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:40:56,690 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:40:59,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:02,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:07,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:09,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:11,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:41:11,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:41:12,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:18,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:19,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:23,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:41:26,565 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:41:26,764 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:41:29,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:29,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:35,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:39,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:41,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:41,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:41:41,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:41:47,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:49,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:41:53,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:56,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:41:56,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:41:56,780 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:41:59,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:41:59,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:04,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:05,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:42:05,634 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:42:05,634 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:42:05,635 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:42:06,448 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:42:08,449 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:42:09,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:09,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:11,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:42:11,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:42:15,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:19,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:20,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:26,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:42:26,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:42:26,732 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:42:29,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:32,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:38,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:41,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:42:41,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:42:44,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:49,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:49,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:42:55,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:42:56,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:42:56,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:42:56,729 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:42:59,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:00,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:05,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:09,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:11,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:11,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:43:11,567 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:43:16,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:19,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:22,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:26,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:43:26,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:43:26,716 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:43:27,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:29,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:33,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:38,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:39,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:41,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:43:41,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:43:43,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:48,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:49,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:43:54,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:43:56,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:43:56,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:43:56,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:43:59,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:00,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:05,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:08,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:44:08,450 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:44:08,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:44:08,451 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:44:08,496 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:44:09,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:10,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:10,497 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:44:11,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:44:11,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:44:16,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:19,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:21,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:26,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:44:26,569 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:44:26,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:44:26,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:29,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:32,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:38,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:39,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:41,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:44:41,569 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:44:44,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:49,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:49,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:44:55,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:44:56,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:44:56,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:44:56,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:44:59,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:00,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:06,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:09,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:11,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:11,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:45:11,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:45:17,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:19,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:22,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:26,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:45:26,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:45:26,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:45:27,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:29,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:33,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:39,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:39,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:41,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:45:41,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:45:44,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:49,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:49,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:45:55,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:45:56,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:45:56,571 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:45:56,723 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:45:59,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:01,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:06,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:09,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:09,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:46:09,652 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:46:09,652 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:46:09,653 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:46:10,536 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:46:11,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:46:11,571 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:46:11,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:12,537 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:46:17,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:19,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:23,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:26,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:46:26,571 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:46:26,700 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:46:28,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:29,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:33,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:39,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:39,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:41,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:46:41,571 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:46:45,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:49,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:46:50,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:56,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:46:56,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:46:56,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:46:56,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:46:59,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:02,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:07,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:09,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:11,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:47:11,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:47:12,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:18,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:19,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:23,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:26,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:47:26,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:47:26,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:47:29,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:29,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:35,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:39,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:40,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:41,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:47:41,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:47:46,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:49,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:47:51,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:56,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:47:56,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:47:56,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:47:56,752 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:47:59,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:01,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:07,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:09,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:10,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:48:10,989 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:48:10,989 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:48:10,991 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:48:11,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:48:11,576 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:48:11,581 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:48:12,581 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:48:12,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:18,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:19,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:24,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:26,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:48:26,578 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:48:26,706 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:48:29,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:29,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:35,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:39,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:41,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:41,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:48:41,579 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:48:47,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:49,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:48:52,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:56,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:48:56,580 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:48:56,726 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:48:57,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:48:59,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:03,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:08,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:09,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:11,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:49:11,579 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:49:14,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:19,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:19,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:25,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:26,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:49:26,580 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:49:26,778 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:49:29,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:30,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:36,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:39,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:41,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:41,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:49:41,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:49:47,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:49,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:49:52,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:56,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:49:56,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:49:56,788 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:49:57,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:49:59,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:03,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:09,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:09,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:11,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:50:11,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:50:14,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:50:14,040 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:50:14,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:50:14,041 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:50:14,629 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:50:15,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:16,631 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:50:19,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:20,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:26,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:26,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:50:26,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:50:26,745 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:50:29,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:31,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:37,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:39,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:41,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:50:41,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:50:42,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:48,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:49,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:50:54,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:50:56,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:50:56,581 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:50:56,706 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:50:59,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:00,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:05,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:09,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:11,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:11,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:51:11,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:51:17,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:19,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:22,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:26,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:51:26,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:51:26,797 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:51:27,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:29,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:33,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:38,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:39,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:41,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:51:41,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:51:44,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:49,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:51:49,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:55,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:51:56,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:51:56,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:51:56,755 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:51:59,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:00,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:06,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:09,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:11,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:11,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:52:11,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:52:15,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:52:15,500 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:52:15,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:52:15,502 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:52:15,682 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:52:16,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:16,683 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:52:19,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:22,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:26,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:52:26,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:52:26,724 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:52:27,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:29,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:33,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:38,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:39,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:41,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:52:41,585 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:52:44,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:49,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:52:50,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:56,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:52:56,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:52:56,583 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:52:56,720 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:52:59,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:01,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:06,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:09,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:11,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:11,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:53:11,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:53:17,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:19,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:23,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:26,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:53:26,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:53:26,750 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:53:28,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:29,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:34,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:39,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:39,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:41,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:53:41,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:53:45,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:49,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:53:50,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:56,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:53:56,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:53:56,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:53:56,722 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:53:59,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:01,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:07,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:09,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:11,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:54:11,585 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:54:12,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:16,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:54:16,961 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:54:16,961 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:54:16,964 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:54:17,735 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:54:17,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:18,736 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:54:19,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:23,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:26,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:54:26,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:54:26,728 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:54:29,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:29,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:34,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:39,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:40,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:41,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:54:41,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:54:46,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:49,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:54:52,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:56,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:54:56,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:54:56,753 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:54:57,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:54:59,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:03,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:09,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:09,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:11,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:55:11,587 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:55:15,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:19,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:21,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:26,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:26,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:55:26,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:55:26,716 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:55:29,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:32,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:37,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:39,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:41,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:55:41,587 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:55:42,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:48,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:49,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:54,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:55:56,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:55:56,587 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:55:56,758 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:55:59,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:55:59,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:04,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:09,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:56:10,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:11,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:56:11,587 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:56:15,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:18,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:56:18,362 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:56:18,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:56:18,364 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:56:18,789 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:56:19,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:56:20,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:20,790 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:56:26,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:26,654 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:56:27,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:56:27,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:56:29,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:56:31,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:37,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:39,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:56:42,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:56:42,963 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:56:43,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:48,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:49,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:56:53,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:56,657 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:56:57,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:56:57,963 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:56:58,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:56:59,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:04,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:09,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:09,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:12,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:57:12,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:57:15,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:19,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:21,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:26,659 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:57:26,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:27,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:57:27,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:57:29,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:32,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:37,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:39,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:42,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:57:42,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:57:43,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:48,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:49,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:54,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:57:56,661 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:57:57,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:57:57,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:57:59,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:57:59,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:04,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:09,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:58:10,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:12,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:58:12,966 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:58:16,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:19,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:58:21,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 01:58:21,271 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 01:58:21,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 01:58:21,272 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 01:58:21,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 01:58:22,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:22,831 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 01:58:26,663 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:58:27,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:27,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:58:27,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:58:29,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:58:33,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:38,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:39,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:58:42,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:58:42,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:58:44,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:49,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:58:49,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:54,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:58:56,670 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:58:57,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:58:57,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:58:59,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:00,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:06,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:09,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:11,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:12,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:59:12,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:59:16,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:19,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:21,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:26,668 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:59:26,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:27,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:59:27,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:59:29,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:32,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:38,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:39,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:42,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:59:42,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:59:43,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:48,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:49,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:53,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 01:59:56,671 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 01:59:57,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 01:59:57,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 01:59:59,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 01:59:59,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:04,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:09,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:10,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:12,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:00:12,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:00:16,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:19,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:22,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:00:22,398 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:00:22,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:00:22,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:22,400 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:00:22,877 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:00:24,879 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:00:26,672 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:00:27,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:27,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:00:27,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:00:29,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:33,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:38,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:39,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:42,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:00:42,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:00:44,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:49,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:49,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:54,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:00:56,675 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:00:57,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:00:57,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:00:59,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:00:59,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:05,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:09,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:10,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:12,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:01:12,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:01:16,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:19,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:21,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:26,678 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:01:26,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:27,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:01:27,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:01:29,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:32,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:38,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:39,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:42,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:01:42,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:01:43,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:48,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:49,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:53,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:01:56,680 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:01:57,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:01:57,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:01:59,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:01:59,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:05,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:09,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:02:10,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:12,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:02:12,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:02:15,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:19,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:02:21,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:23,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:02:23,796 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:02:23,796 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:02:23,797 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:02:23,936 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:02:25,938 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:02:26,682 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:02:26,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:27,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:02:27,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:02:29,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:02:32,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:37,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:39,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:02:42,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:42,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:02:42,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:02:48,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:49,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:02:53,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:56,684 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:02:57,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:02:57,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:02:58,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:02:59,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:04,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:09,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:10,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:12,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:03:12,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:03:16,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:19,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:22,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:26,690 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:03:27,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:27,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:03:27,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:03:29,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:33,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:38,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:39,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:42,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:03:42,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:03:43,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:48,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:49,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:53,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:03:56,689 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:03:57,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:03:57,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:03:59,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:03:59,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:04,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:09,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:04:10,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:12,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:04:12,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:04:16,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:19,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:04:22,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:26,693 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:04:26,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:04:26,897 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:04:26,897 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:04:26,898 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:04:26,996 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:04:27,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:27,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:04:27,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:04:27,997 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:04:29,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:04:33,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:38,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:39,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:04:42,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:04:42,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:04:43,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:49,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:04:49,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:54,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:04:56,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:04:57,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:04:57,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:04:59,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:00,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:05,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:09,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:11,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:12,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:05:12,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:05:17,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:19,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:23,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:26,696 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:05:27,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:05:27,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:05:28,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:29,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:34,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:39,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:40,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:42,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:05:42,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:05:46,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:49,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:05:52,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:56,698 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:05:57,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:05:57,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:05:57,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:05:59,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:03,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:09,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:09,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:12,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:06:12,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:06:14,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:19,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:20,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:26,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:26,702 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:06:27,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:06:27,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:06:28,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:06:28,402 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:06:28,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:06:28,403 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:06:29,056 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:06:29,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:30,057 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:06:32,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:37,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:39,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:42,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:06:42,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:06:43,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:48,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:49,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:54,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:06:56,703 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:06:57,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:06:57,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:06:59,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:06:59,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:05,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:09,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:07:11,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:12,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:07:12,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:07:16,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:19,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:07:22,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:26,705 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:07:27,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:27,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:07:27,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:07:29,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:07:33,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:39,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:07:39,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:42,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:07:42,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:07:45,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:49,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:07:50,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:56,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:07:56,710 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:07:57,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:07:57,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:07:59,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:02,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:07,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:09,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:12,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:08:12,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:08:13,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:18,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:19,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:24,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:26,710 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:08:27,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:08:27,977 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:08:29,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:29,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:08:29,832 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:08:29,832 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:08:29,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:29,834 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:08:30,117 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:08:30,117 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:08:35,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:39,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:40,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:42,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:08:42,977 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:08:46,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:49,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:08:51,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:56,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:08:56,713 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:08:57,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:08:57,977 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:08:59,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:02,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:07,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:09,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:12,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:09:12,977 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:09:13,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:18,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:20,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:24,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:26,714 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:09:27,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:09:27,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:09:30,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:30,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:35,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:40,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:41,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:42,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:09:42,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:09:46,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:50,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:09:52,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:56,717 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:09:57,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:09:57,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:09:57,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:10:00,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:03,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:08,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:10,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:12,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:10:12,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:10:13,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:19,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:20,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:24,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:26,719 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:10:27,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:10:27,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:10:30,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:30,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:32,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:10:32,842 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:10:32,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:10:32,843 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:10:33,174 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:10:34,175 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:10:36,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:40,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:41,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:42,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:10:42,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:10:47,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:50,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:10:52,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:56,722 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:10:57,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:10:57,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:10:57,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:11:00,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:03,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:08,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:10,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:12,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:11:12,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:11:14,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:19,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:20,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:25,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:26,724 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:11:27,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:11:27,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:11:30,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:31,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:36,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:40,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:41,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:42,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:11:42,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:11:46,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:50,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:11:52,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:56,726 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:11:57,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:11:57,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:11:57,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:12:00,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:03,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:08,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:10,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:12,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:12:12,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:12:14,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:20,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:20,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:26,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:26,728 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:12:27,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:12:27,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:12:30,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:32,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:34,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:12:34,347 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:12:34,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:12:34,347 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:12:35,235 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:12:36,236 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:12:38,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:40,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:42,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:12:42,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:12:43,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:48,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:50,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:12:53,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:12:56,731 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:12:57,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:12:57,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:12:59,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:00,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:05,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:10,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:11,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:12,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:13:12,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:13:16,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:20,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:22,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:26,733 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:13:27,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:27,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:13:27,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:13:30,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:33,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:38,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:40,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:42,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:13:42,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:13:43,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:49,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:50,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:13:55,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:13:56,735 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:13:57,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:13:57,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:14:00,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:01,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:06,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:10,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:12,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:12,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:14:12,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:14:18,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:20,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:24,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:26,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:14:27,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:14:27,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:14:30,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:30,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:35,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:14:35,801 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:14:35,801 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:14:35,802 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:14:35,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:36,286 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:14:36,287 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:14:40,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:41,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:42,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:14:42,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:14:46,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:50,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:14:51,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:56,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:14:56,740 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:14:57,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:14:57,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:15:00,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:02,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:08,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:10,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:12,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:15:12,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:15:14,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:19,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:20,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:25,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:26,742 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:15:27,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:15:27,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:15:30,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:31,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:36,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:40,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:42,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:42,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:15:42,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:15:47,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:50,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:15:52,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:56,744 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:15:57,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:15:57,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:15:57,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:16:00,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:03,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:09,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:10,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:12,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:16:12,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:16:14,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:19,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:20,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:25,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:26,746 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:16:27,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:16:27,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:16:30,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:30,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:36,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:38,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:16:38,921 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:16:38,921 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:16:38,922 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:16:39,327 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:16:40,328 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:16:40,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:41,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:42,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:16:42,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:16:47,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:50,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:16:53,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:16:56,748 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:16:57,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:16:57,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:16:59,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:00,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:05,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:10,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:10,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:12,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:17:12,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:17:16,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:20,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:21,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:26,755 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:17:26,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:27,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:17:27,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:17:30,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:32,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:37,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:40,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:42,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:42,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:17:42,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:17:48,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:50,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:17:53,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:17:56,753 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:17:57,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:17:57,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:17:59,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:00,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:04,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:10,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:10,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:12,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:18:12,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:18:15,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:20,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:21,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:26,757 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:18:26,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:27,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:18:27,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:18:30,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:32,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:38,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:40,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:18:40,503 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:18:40,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:18:40,504 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:18:40,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:41,381 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:18:42,383 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:18:42,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:18:42,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:18:43,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:49,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:50,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:18:54,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:18:56,758 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:18:57,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:18:57,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:19:00,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:00,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:05,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:10,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:11,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:12,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:19:12,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:19:16,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:20,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:22,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:26,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:19:27,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:28,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:19:28,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:19:30,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:33,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:38,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:40,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:43,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:19:43,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:19:44,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:49,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:50,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:19:55,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:19:56,762 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:19:58,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:19:58,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:20:00,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:00,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:05,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:10,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:11,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:13,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:20:13,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:20:17,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:20,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:22,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:26,765 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:20:27,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:28,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:20:28,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:20:30,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:33,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:39,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:40,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:42,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:20:42,151 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:20:42,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:20:42,152 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:20:42,428 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:20:43,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:20:43,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:20:44,429 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:20:44,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:49,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:50,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:20:55,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:20:56,767 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:20:58,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:20:58,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:21:00,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:00,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:05,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:10,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:10,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:13,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:21:13,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:21:16,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:20,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:22,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:26,770 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:21:27,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:28,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:21:28,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:21:30,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:33,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:38,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:40,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:43,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:21:43,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:21:44,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:49,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:50,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:21:54,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:21:56,775 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:21:58,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:21:58,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:22:00,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:00,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:06,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:10,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:11,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:13,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:22:13,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:22:16,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:20,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:21,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:26,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:26,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:22:28,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:22:28,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:22:30,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:32,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:37,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:40,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:43,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:22:43,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:22:43,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:44,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:22:44,970 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:22:44,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:22:44,972 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:22:45,480 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:22:46,481 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:22:48,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:50,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:22:54,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:22:56,776 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:22:58,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:22:58,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:23:00,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:00,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:05,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:10,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:11,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:13,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:23:13,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:23:16,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:20,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:22,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:26,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:23:27,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:28,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:23:28,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:23:30,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:33,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:38,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:40,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:43,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:23:43,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:23:44,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:50,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:50,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:23:56,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:23:56,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:23:58,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:23:58,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:24:00,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:01,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:07,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:10,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:12,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:13,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:24:13,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:24:18,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:20,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:24,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:26,783 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:24:28,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:24:28,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:24:30,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:30,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:36,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:40,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:42,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:43,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:24:43,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:24:46,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:24:46,425 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:24:46,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:24:46,426 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:24:46,532 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:24:48,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:48,533 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:24:50,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:24:53,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:24:56,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:24:58,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:24:58,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:24:58,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:00,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:04,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:10,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:10,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:13,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:25:13,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:25:16,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:20,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:21,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:26,788 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:25:26,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:28,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:25:28,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:25:30,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:32,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:37,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:40,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:42,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:43,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:25:43,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:25:47,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:50,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:25:53,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:25:56,790 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:25:58,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:25:58,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:25:59,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:00,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:05,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:10,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:11,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:13,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:26:13,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:26:16,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:20,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:22,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:26,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:26:27,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:28,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:26:28,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:26:30,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:33,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:38,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:40,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:43,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:26:43,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:26:43,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:47,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:26:47,815 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:26:47,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:26:47,816 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:26:48,583 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:26:48,584 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:26:49,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:50,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:26:55,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:26:56,794 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:26:58,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:26:58,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:27:00,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:00,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:05,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:10,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:11,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:13,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:27:13,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:27:17,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:20,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:22,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:26,799 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:27:27,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:28,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:27:28,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:27:30,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:33,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:38,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:40,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:43,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:27:43,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:27:44,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:49,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:50,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:27:54,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:27:56,799 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:27:58,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:27:58,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:28:00,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:00,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:05,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:10,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:11,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:13,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:28:13,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:28:16,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:20,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:21,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:26,801 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:28:26,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:28,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:28:28,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:28:30,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:32,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:37,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:40,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:43,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:28:43,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:28:43,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:48,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:50,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:28:50,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:28:50,978 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:28:50,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:28:50,979 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:28:51,633 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:28:52,634 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:28:53,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:28:56,803 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:28:58,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:28:58,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:28:59,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:00,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:04,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:09,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:10,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:13,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:29:13,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:29:15,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:20,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:21,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:26,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:26,805 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:29:28,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:29:28,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:29:30,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:32,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:37,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:40,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:43,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:29:43,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:29:43,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:48,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:50,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:29:53,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:29:56,807 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:29:58,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:29:58,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:29:59,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:00,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:04,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:09,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:10,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:13,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:30:13,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:30:15,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:20,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:20,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:26,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:26,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:30:28,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:30:28,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:30:30,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:32,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:37,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:40,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:43,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:30:43,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:30:43,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:48,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:50,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:30:52,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:30:52,337 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:30:52,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:30:52,338 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:30:52,685 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:30:54,687 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:30:54,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:30:56,815 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:30:58,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:30:58,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:30:59,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:00,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:05,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:10,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:10,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:13,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:31:13,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:31:15,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:20,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:20,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:25,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:26,814 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:31:28,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:31:28,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:31:30,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:30,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:35,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:40,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:40,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:43,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:31:43,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:31:46,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:50,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:31:51,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:56,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:31:56,816 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:31:58,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:31:58,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:32:00,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:02,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:07,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:10,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:13,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:32:13,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:32:13,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:18,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:20,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:24,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:26,818 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:32:28,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:32:28,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:32:29,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:30,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:35,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:40,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:40,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:43,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:32:43,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:32:45,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:50,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:32:51,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:53,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:32:53,780 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:32:53,780 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:32:53,781 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:32:54,748 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:32:54,749 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:32:56,821 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:32:56,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:32:58,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:32:58,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:33:00,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:02,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:08,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:10,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:13,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:33:13,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:33:14,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:19,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:20,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:25,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:26,823 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:33:28,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:33:28,020 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:33:30,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:31,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:36,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:40,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:42,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:43,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:33:43,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:33:48,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:50,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:33:54,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:33:56,825 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:33:58,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:33:58,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:34:00,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:00,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:05,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:10,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:11,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:13,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:34:13,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:34:16,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:20,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:21,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:26,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:34:26,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:28,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:34:28,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:34:30,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:32,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:37,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:40,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:43,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:34:43,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:34:43,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:48,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:50,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:34:54,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:34:55,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:34:55,197 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:34:55,197 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:34:55,198 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:34:55,793 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:34:56,793 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:34:56,829 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:34:58,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:34:58,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:34:59,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:00,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:05,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:10,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:10,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:13,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:35:13,022 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:35:15,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:20,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:21,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:26,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:26,831 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:35:28,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:35:28,022 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:35:30,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:32,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:37,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:40,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:42,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:43,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:35:43,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:35:47,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:50,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:35:53,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:35:56,839 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:35:58,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:35:58,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:35:58,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:00,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:04,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:09,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:10,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:13,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:36:13,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:36:15,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:20,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:20,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:25,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:26,836 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:36:28,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:36:28,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:36:30,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:30,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:36,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:40,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:41,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:43,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:36:43,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:36:46,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:50,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:36:52,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:56,840 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:36:57,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:36:58,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:36:58,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:36:58,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:36:58,208 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:36:58,209 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:36:58,210 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:36:58,843 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:36:58,843 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:37:00,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:03,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:09,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:10,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:13,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:37:13,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:37:14,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:20,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:20,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:25,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:26,841 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:37:28,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:37:28,025 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:37:30,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:30,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:36,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:40,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:41,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:43,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:37:43,025 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:37:46,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:50,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:37:51,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:56,843 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:37:56,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:37:58,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:37:58,025 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:38:00,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:02,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:07,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:10,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:13,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:38:13,025 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:38:13,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:18,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:20,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:23,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:26,845 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:38:28,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:38:28,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:38:28,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:30,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:34,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:40,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:40,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:43,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:38:43,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:38:45,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:50,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:38:51,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:56,847 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:38:56,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:38:58,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:38:58,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:38:59,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:38:59,688 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:38:59,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:38:59,689 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:38:59,897 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:39:00,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:00,897 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:39:02,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:07,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:10,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:13,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:39:13,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:39:13,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:18,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:20,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:23,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:26,849 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:39:28,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:39:28,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:39:28,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:30,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:33,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:39,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:40,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:43,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:39:43,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:39:45,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:50,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:50,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:39:56,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:39:56,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:39:58,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:39:58,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:40:00,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:02,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:07,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:10,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:12,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:13,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:40:13,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:40:18,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:20,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:24,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:26,857 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:40:28,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:40:28,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:40:30,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:30,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:35,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:40,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:41,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:43,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:40:43,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:40:46,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:50,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:40:51,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:56,856 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:40:56,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:40:58,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:40:58,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:41:00,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:01,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:41:01,079 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:41:01,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:41:01,081 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:41:01,939 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:41:02,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:02,939 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:41:07,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:13,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:41:13,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:41:13,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:18,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:20,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:23,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:26,859 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:41:28,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:41:28,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:41:29,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:30,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:34,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:40,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:40,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:43,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:41:43,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:41:46,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:50,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:41:52,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:56,862 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:41:57,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:41:58,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:41:58,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:42:00,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:03,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:09,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:10,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:13,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:42:13,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:42:15,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:20,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:20,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:25,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:26,863 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:42:28,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:42:28,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:42:30,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:31,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:37,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:40,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:43,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:42:43,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:42:43,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:48,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:50,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:42:54,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:42:56,866 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:42:58,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:42:58,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:42:59,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:00,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:04,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:43:04,046 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:43:04,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:43:04,048 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:43:04,991 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:43:04,991 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:43:05,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:10,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:10,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:13,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:43:13,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:43:16,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:20,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:21,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:26,931 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:43:26,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:28,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:43:28,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:43:30,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:32,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:38,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:40,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:43,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:43:43,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:43:44,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:49,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:50,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:43:55,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:43:56,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:43:58,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:43:58,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:44:00,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:00,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:06,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:10,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:12,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:13,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:44:13,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:44:17,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:20,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:23,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:26,935 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:44:28,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:44:28,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:44:29,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:30,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:34,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:40,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:40,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:43,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:44:43,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:44:46,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:50,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:44:52,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:56,941 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:44:57,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:44:58,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:44:58,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:45:00,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:03,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:05,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:45:05,337 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:45:05,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:45:05,339 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:45:06,039 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:45:07,039 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:45:09,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:10,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:13,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:45:13,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:45:15,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:20,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:21,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:26,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:26,939 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:45:28,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:45:28,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:45:30,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:32,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:37,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:40,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:43,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:45:43,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:45:43,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:48,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:50,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:45:53,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:45:56,943 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:45:58,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:45:58,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:45:59,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:00,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:05,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:10,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:10,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:13,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:46:13,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:46:15,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:20,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:21,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:26,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:26,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:46:28,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:46:28,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:46:30,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:32,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:37,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:40,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:42,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:43,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:46:43,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:46:48,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:50,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:46:53,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:46:56,946 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:46:58,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:46:58,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:46:58,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:00,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:04,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:06,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:47:06,743 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:47:06,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:47:06,745 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:47:07,079 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:47:09,079 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:47:10,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:10,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:13,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:47:13,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:47:15,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:20,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:21,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:26,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:26,948 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:47:28,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:47:28,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:47:30,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:31,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:37,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:40,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:42,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:43,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:47:43,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:47:47,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:50,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:47:53,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:47:56,951 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:47:58,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:47:58,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:47:59,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:00,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:05,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:10,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:11,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:13,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:48:13,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:48:16,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:20,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:21,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:26,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:26,953 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:48:28,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:48:28,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:48:30,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:32,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:37,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:40,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:43,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:48:43,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:48:43,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:48,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:50,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:48:54,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:48:56,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:48:58,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:48:58,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:48:59,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:00,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:05,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:09,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:49:09,890 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:49:09,890 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:49:09,891 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:49:10,127 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:49:10,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:10,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:11,127 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:49:13,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:49:13,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:49:16,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:20,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:22,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:26,962 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:49:27,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:28,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:49:28,039 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:49:30,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:33,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:39,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:40,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:43,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:49:43,039 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:49:45,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:50,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:49:51,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:56,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:49:56,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:49:58,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:49:58,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:50:00,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:02,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:08,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:10,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:13,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:50:13,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:50:13,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:19,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:20,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:25,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:26,964 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:50:28,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:50:28,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:50:30,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:30,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:35,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:40,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:41,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:43,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:50:43,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:50:47,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:50,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:50:52,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:56,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:50:57,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:50:58,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:50:58,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:51:00,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:03,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:08,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:10,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:11,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:51:11,214 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:51:11,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:51:11,215 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:51:12,169 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:51:13,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:51:13,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:51:13,171 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:51:14,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:19,175 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:51:20,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:20,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:25,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:26,969 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:51:28,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:51:28,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:51:29,182 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:51:30,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:30,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:36,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:40,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:41,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:43,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:51:43,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:51:47,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:50,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:51:53,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:51:56,972 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:51:58,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:51:58,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:51:59,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:00,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:04,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:09,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:10,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:13,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:52:13,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:52:15,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:20,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:20,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:25,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:26,975 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:52:28,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:52:28,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:52:30,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:31,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:36,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:40,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:42,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:43,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:52:43,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:52:48,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:50,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:52:54,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:52:56,976 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:52:58,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:52:58,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:52:59,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:00,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:04,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:10,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:10,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:13,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:53:13,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:53:16,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:20,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:21,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:26,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:26,979 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:53:27,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:53:27,364 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:53:27,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:53:27,365 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:53:28,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:53:28,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:53:28,230 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:53:29,231 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:53:30,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:32,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:38,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:40,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:43,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:53:43,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:53:43,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:48,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:50,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:53:53,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:53:56,984 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:53:58,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:53:58,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:53:58,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:00,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:03,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:08,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:10,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:13,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:54:13,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:54:14,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:19,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:20,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:24,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:26,983 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:54:28,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:54:28,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:54:30,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:30,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:36,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:40,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:42,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:43,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:54:43,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:54:48,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:50,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:54:54,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:54:56,987 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:54:58,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:54:58,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:55:00,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:00,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:05,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:10,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:10,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:13,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:55:13,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:55:16,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:20,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:21,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:26,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:26,988 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:55:28,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:55:28,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:55:30,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:55:30,553 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:55:30,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:55:30,554 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:55:30,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:31,274 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:55:31,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:33,275 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:55:37,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:40,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:42,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:43,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:55:43,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:55:48,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:50,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:55:54,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:55:56,990 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:55:58,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:55:58,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:55:59,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:00,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:05,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:10,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:11,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:13,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:56:13,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:56:16,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:20,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:22,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:26,992 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:56:27,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:28,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:56:28,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:56:30,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:33,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:38,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:40,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:43,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:56:43,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:56:44,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:49,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:50,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:56:54,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:56:56,994 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:56:58,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:56:58,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:57:00,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:00,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:05,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:10,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:10,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:13,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:57:13,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:57:16,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:20,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:22,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:26,997 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:57:27,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:28,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:57:28,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:57:30,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:32,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:57:32,025 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:57:32,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:57:32,027 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:57:32,322 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:57:33,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:33,323 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:57:38,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:40,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:43,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:57:43,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:57:44,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:49,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:50,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:57:55,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:57:56,998 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:57:58,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:57:58,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:58:00,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:00,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:06,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:10,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:11,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:13,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:58:13,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:58:17,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:20,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:23,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:27,004 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:58:28,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:58:28,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:58:29,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:30,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:34,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:40,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:40,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:43,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:58:43,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:58:46,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:50,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:58:51,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:57,003 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:58:57,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:58:58,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:58:58,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:59:00,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:02,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:07,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:10,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:13,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:59:13,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:59:13,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:18,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:20,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:23,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:27,007 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:59:28,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:59:28,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:59:29,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:30,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:33,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 02:59:33,594 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 02:59:33,594 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 02:59:33,595 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 02:59:34,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:34,376 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 02:59:35,377 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 02:59:39,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:40,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:43,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:59:43,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 02:59:45,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:50,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 02:59:51,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:57,008 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 02:59:57,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 02:59:58,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 02:59:58,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:00:00,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:02,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:08,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:10,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:13,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:00:13,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:00:14,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:19,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:20,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:25,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:27,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:00:28,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:00:28,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:00:30,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:31,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:37,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:40,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:42,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:43,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:00:43,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:00:47,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:50,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:00:53,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:00:57,012 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:00:58,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:00:58,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:00:59,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:00,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:04,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:10,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:10,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:13,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:01:13,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:01:16,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:20,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:22,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:27,015 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:01:28,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:28,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:01:28,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:01:30,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:33,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:36,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:01:36,527 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:01:36,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:01:36,528 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:01:37,434 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:01:38,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:39,435 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:01:40,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:43,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:01:43,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:01:44,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:49,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:50,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:01:54,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:01:57,017 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:01:58,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:01:58,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:02:00,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:00,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:05,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:10,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:11,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:13,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:02:13,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:02:17,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:20,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:22,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:27,019 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:02:28,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:28,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:02:28,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:02:30,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:33,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:38,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:40,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:43,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:02:43,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:02:44,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:49,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:50,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:02:55,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:02:57,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:02:58,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:02:58,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:03:00,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:01,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:06,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:10,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:12,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:13,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:03:13,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:03:17,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:20,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:23,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:27,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:03:28,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:03:28,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:03:29,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:30,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:34,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:37,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:03:37,885 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:03:37,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:03:37,886 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:03:38,487 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:03:39,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:39,488 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:03:40,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:43,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:03:43,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:03:45,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:50,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:50,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:03:55,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:03:57,026 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:03:58,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:03:58,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:04:00,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:00,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:05,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:10,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:11,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:13,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:04:13,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:04:16,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:20,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:22,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:27,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:04:28,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:28,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:04:28,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:04:30,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:33,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:39,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:40,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:43,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:04:43,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:04:45,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:50,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:50,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:04:56,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:04:57,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:04:58,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:04:58,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:05:00,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:02,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:07,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:10,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:13,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:05:13,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:05:13,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:18,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:20,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:23,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:27,032 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:05:28,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:05:28,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:05:29,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:30,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:34,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:39,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:05:39,335 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:05:39,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:05:39,336 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:05:39,537 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:05:40,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:40,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:41,537 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:05:43,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:05:43,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:05:45,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:50,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:05:51,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:57,035 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:05:57,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:05:58,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:05:58,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:06:00,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:02,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:07,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:10,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:12,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:13,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:06:13,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:06:18,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:20,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:24,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:27,037 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:06:28,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:06:28,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:06:29,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:30,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:34,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:39,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:40,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:43,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:06:43,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:06:44,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:49,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:50,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:06:54,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:06:57,039 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:06:58,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:06:58,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:07:00,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:00,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:06,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:10,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:12,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:13,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:07:13,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:07:18,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:20,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:24,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:27,042 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:07:28,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:07:28,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:07:29,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:30,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:35,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:40,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:07:40,838 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:07:40,839 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:07:40,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:40,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:07:40,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:41,602 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:07:43,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:07:43,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:07:43,603 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:07:46,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:50,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:07:51,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:56,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:07:57,049 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:07:58,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:07:58,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:08:00,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:02,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:07,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:10,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:13,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:08:13,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:08:13,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:18,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:20,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:23,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:27,047 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:08:28,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:08:28,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:08:29,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:30,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:34,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:40,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:40,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:43,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:08:43,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:08:46,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:50,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:08:52,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:57,051 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:08:58,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:08:58,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:08:58,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:09:00,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:03,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:08,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:10,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:13,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:09:13,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:09:14,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:20,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:24,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:27,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:09:28,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:09:28,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:09:30,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:30,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:36,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:40,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:42,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:43,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:09:43,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:09:43,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:09:43,882 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:09:43,882 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:09:43,883 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:09:44,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:09:45,657 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:09:48,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:50,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:09:54,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:09:57,054 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:09:58,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:09:58,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:10:00,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:00,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:05,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:10,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:10,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:13,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:10:13,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:10:16,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:20,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:21,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:27,057 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:10:27,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:28,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:10:28,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:10:30,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:32,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:37,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:40,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:43,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:10:43,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:10:43,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:48,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:50,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:10:54,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:10:57,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:10:58,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:10:58,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:11:00,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:00,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:05,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:10,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:10,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:13,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:11:13,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:11:16,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:20,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:21,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:26,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:27,061 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:11:28,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:11:28,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:11:30,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:32,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:37,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:40,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:43,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:11:43,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:11:43,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:45,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:11:45,391 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:11:45,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:11:45,392 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:11:45,707 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:11:47,707 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:11:48,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:50,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:11:54,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:11:57,064 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:11:58,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:11:58,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:12:00,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:00,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:06,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:10,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:12,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:13,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:12:13,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:12:17,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:20,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:23,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:27,069 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:12:28,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:12:28,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:12:29,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:30,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:34,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:40,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:40,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:43,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:12:43,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:12:46,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:50,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:12:52,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:57,067 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:12:58,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:12:58,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:12:58,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:13:00,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:03,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:09,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:10,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:13,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:13:13,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:13:15,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:20,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:21,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:26,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:27,071 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:13:28,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:13:28,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:13:30,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:32,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:37,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:40,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:43,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:13:43,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:13:43,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:46,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:13:46,976 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:13:46,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:13:46,977 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:13:47,763 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:13:49,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:49,764 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:13:50,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:13:54,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:13:57,072 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:13:58,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:13:58,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:13:59,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:00,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:05,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:10,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:11,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:13,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:14:13,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:14:16,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:20,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:22,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:27,074 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:14:28,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:28,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:14:28,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:14:30,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:33,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:38,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:40,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:43,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:14:43,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:14:44,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:49,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:50,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:14:54,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:14:57,077 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:14:58,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:14:58,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:14:59,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:00,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:04,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:10,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:10,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:13,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:15:13,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:15:16,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:20,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:22,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:27,079 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:15:28,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:15:28,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:15:28,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:30,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:33,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:38,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:40,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:43,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:15:43,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:15:44,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:49,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:49,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:15:49,987 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:15:49,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:15:49,988 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:15:50,808 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:15:50,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:15:51,809 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:15:55,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:15:57,081 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:15:58,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:15:58,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:16:00,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:00,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:06,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:10,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:12,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:13,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:16:13,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:16:17,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:20,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:22,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:27,083 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:16:28,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:16:28,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:16:28,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:30,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:33,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:39,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:40,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:43,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:16:43,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:16:44,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:49,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:50,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:16:55,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:16:57,088 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:16:58,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:16:58,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:17:00,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:00,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:05,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:10,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:11,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:13,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:17:13,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:17:17,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:20,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:22,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:27,088 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:17:28,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:17:28,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:17:28,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:30,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:33,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:39,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:40,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:43,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:17:43,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:17:44,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:49,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:50,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:17:51,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:17:51,544 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:17:51,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:17:51,547 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:17:51,861 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:17:53,862 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:17:54,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:17:57,091 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:17:58,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:17:58,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:18:00,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:00,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:05,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:10,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:11,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:13,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:18:13,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:18:17,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:20,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:23,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:27,094 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:18:28,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:18:28,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:18:29,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:30,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:34,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:39,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:40,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:43,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:18:43,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:18:45,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:50,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:50,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:18:56,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:18:57,095 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:18:58,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:18:58,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:19:00,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:01,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:07,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:10,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:13,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:19:13,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:19:13,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:18,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:20,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:24,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:27,098 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:19:28,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:19:28,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:19:29,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:30,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:34,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:39,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:40,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:43,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:19:43,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:19:44,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:49,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:50,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:19:53,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:19:53,076 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:19:53,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:19:53,077 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:19:53,914 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:19:55,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:19:55,915 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:19:57,100 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:19:58,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:19:58,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:20:00,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:01,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:06,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:10,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:12,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:13,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:20:13,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:20:17,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:20,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:22,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:27,103 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:20:28,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:20:28,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:20:28,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:30,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:33,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:38,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:40,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:43,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:20:43,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:20:44,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:49,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:50,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:20:54,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:20:57,105 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:20:58,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:20:58,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:21:00,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:00,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:05,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:10,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:10,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:13,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:21:13,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:21:16,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:20,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:22,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:27,107 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:21:28,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:21:28,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:21:28,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:30,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:33,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:38,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:40,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:43,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:21:43,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:21:44,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:49,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:50,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:21:54,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:21:56,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:21:56,354 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:21:56,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:21:56,355 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:21:56,971 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:21:57,114 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:21:57,971 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:21:58,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:21:58,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:22:00,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:00,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:06,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:10,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:11,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:13,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:22:13,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:22:17,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:20,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:22,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:27,111 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:22:28,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:22:28,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:22:28,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:30,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:33,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:38,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:40,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:43,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:22:43,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:22:44,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:49,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:50,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:22:54,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:22:57,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:22:58,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:22:58,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:23:00,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:00,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:05,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:10,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:10,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:13,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:23:13,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:23:16,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:20,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:21,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:27,116 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:23:27,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:28,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:23:28,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:23:30,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:32,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:38,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:40,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:43,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:23:43,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:23:44,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:49,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:50,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:23:55,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:23:57,118 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:23:57,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:23:57,954 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:23:57,954 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:23:57,955 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:23:58,031 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:23:58,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:23:58,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:23:59,031 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:24:00,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:00,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:06,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:10,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:11,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:13,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:24:13,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:24:16,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:20,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:21,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:27,121 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:24:27,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:28,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:24:28,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:24:30,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:32,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:37,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:40,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:43,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:24:43,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:24:43,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:48,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:50,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:24:53,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:24:57,123 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:24:58,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:24:58,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:24:59,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:00,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:04,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:09,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:10,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:13,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:25:13,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:25:15,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:20,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:21,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:27,125 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:25:27,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:28,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:25:28,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:25:30,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:32,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:37,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:40,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:42,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:43,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:25:43,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:25:47,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:50,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:25:52,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:57,127 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:25:58,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:25:58,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:25:58,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:25:59,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:25:59,601 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:25:59,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:25:59,602 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:26:00,082 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:26:00,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:01,083 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:26:03,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:09,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:10,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:13,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:26:13,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:26:14,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:19,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:20,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:24,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:27,133 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:26:28,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:26:28,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:26:29,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:30,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:35,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:40,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:40,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:43,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:26:43,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:26:46,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:50,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:26:51,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:56,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:26:57,131 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:26:58,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:26:58,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:27:00,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:02,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:07,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:10,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:13,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:13,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:27:13,599 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:27:19,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:20,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:24,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:27,135 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:27:28,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:27:28,599 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:27:29,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:30,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:35,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:40,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:40,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:43,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:27:43,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:27:46,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:50,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:27:51,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:57,136 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:27:57,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:27:58,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:27:58,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:28:00,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:02,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:02,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:28:02,813 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:28:02,813 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:28:02,814 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:28:03,145 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:28:05,146 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:28:08,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:10,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:13,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:13,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:28:13,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:28:19,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:20,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:24,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:27,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:28:28,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:28:28,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:28:29,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:30,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:35,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:40,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:41,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:43,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:28:43,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:28:46,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:50,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:28:52,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:57,141 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:28:58,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:28:58,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:28:58,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:29:00,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:03,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:08,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:10,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:13,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:29:13,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:29:13,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:19,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:20,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:25,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:27,143 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:29:28,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:29:28,601 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:29:30,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:31,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:37,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:40,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:43,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:43,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:29:43,601 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:29:48,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:50,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:29:53,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:29:57,145 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:29:58,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:29:58,601 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:29:58,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:00,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:04,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:30:04,542 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:30:04,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:30:04,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:04,543 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:30:05,200 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:30:05,201 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:30:09,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:10,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:13,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:30:13,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:30:15,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:20,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:20,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:26,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:27,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:30:28,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:30:28,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:30:30,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:32,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:37,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:40,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:43,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:43,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:30:43,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:30:48,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:50,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:30:53,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:30:57,153 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:30:58,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:30:58,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:30:58,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:00,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:04,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:10,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:10,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:13,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:31:13,602 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:31:15,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:20,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:21,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:27,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:31:27,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:28,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:31:28,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:31:30,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:32,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:38,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:40,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:43,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:31:43,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:31:43,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:49,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:50,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:31:54,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:31:57,156 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:31:58,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:31:58,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:32:00,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:00,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:06,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:32:06,190 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:32:06,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:32:06,191 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:32:06,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:06,248 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:32:07,249 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:32:10,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:11,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:13,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:32:13,603 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:32:16,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:20,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:22,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:27,157 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:32:28,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:28,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:32:28,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:32:30,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:33,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:39,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:40,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:43,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:32:43,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:32:44,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:49,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:50,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:32:55,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:32:57,159 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:32:58,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:32:58,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:33:00,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:01,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:06,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:10,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:12,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:13,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:33:13,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:33:17,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:20,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:23,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:27,162 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:33:28,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:33:28,605 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:33:28,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:30,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:34,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:39,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:40,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:43,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:33:43,605 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:33:44,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:50,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:50,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:33:55,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:33:57,165 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:33:58,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:33:58,605 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:34:00,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:00,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:05,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:09,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:34:09,267 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:34:09,267 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:34:09,268 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:34:09,298 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:34:10,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:11,298 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:34:11,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:13,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:34:13,606 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:34:16,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:20,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:21,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:27,167 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:34:27,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:28,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:34:28,607 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:34:30,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:32,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:37,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:40,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:43,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:43,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:34:43,606 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:34:48,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:50,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:34:54,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:34:57,169 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:34:58,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:34:58,608 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:34:59,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:00,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:04,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:10,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:10,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:13,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:35:13,609 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:35:16,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:21,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:21,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:27,175 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:35:27,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:28,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:35:28,609 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:35:31,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:32,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:37,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:41,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:43,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:43,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:35:43,609 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:35:48,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:51,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:35:54,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:35:57,174 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:35:58,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:35:58,610 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:35:59,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:01,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:05,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:10,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:36:10,853 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:36:10,854 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:36:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:10,856 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:36:11,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:11,352 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:36:13,352 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:36:13,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:36:13,610 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:36:16,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:21,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:22,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:27,177 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:36:28,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:28,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:36:28,610 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:36:31,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:33,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:38,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:41,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:43,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:36:43,610 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:36:43,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:49,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:51,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:36:54,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:36:57,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:36:58,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:36:58,610 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:37:00,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:01,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:06,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:11,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:12,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:13,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:37:13,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:37:17,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:21,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:23,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:27,181 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:37:28,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:37:28,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:37:28,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:31,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:34,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:39,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:41,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:43,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:37:43,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:37:45,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:50,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:51,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:37:55,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:37:57,183 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:37:58,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:37:58,612 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:38:00,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:01,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:06,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:11,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:12,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:38:12,356 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:38:12,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:38:12,357 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:38:12,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:12,399 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:38:13,399 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:38:13,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:38:13,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:38:17,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:21,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:23,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:27,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:38:28,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:38:28,612 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:38:28,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:31,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:34,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:40,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:41,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:43,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:38:43,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:38:45,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:51,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:38:51,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:57,188 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:38:57,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:38:58,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:38:58,612 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:39:01,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:02,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:08,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:11,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:13,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:39:13,613 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:39:13,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:19,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:21,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:25,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:27,190 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:39:28,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:39:28,613 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:39:31,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:31,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:37,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:41,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:43,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:43,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:39:43,612 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:39:49,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:51,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:39:55,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:39:57,193 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:39:58,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:39:58,613 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:40:01,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:01,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:06,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:11,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:12,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:13,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:40:13,613 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:40:15,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:40:15,466 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:40:15,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:40:15,467 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:40:16,449 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:40:17,449 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:40:18,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:21,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:23,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:27,199 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:40:28,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:40:28,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:40:28,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:31,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:34,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:39,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:41,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:43,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:40:43,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:40:45,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:50,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:51,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:40:56,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:40:57,197 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:40:58,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:40:58,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:41:01,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:01,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:07,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:11,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:13,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:13,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:41:13,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:41:18,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:21,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:24,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:27,201 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:41:28,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:41:28,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:41:29,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:31,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:34,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:39,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:41,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:43,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:41:43,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:41:44,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:49,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:51,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:41:54,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:41:57,202 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:41:58,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:41:58,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:42:00,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:01,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:05,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:11,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:11,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:13,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:42:13,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:42:17,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:42:17,063 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:42:17,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:42:17,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:17,065 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:42:17,500 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:42:19,502 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:42:21,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:22,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:27,204 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:42:28,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:28,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:42:28,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:42:31,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:33,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:38,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:41,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:43,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:42:43,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:42:43,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:49,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:51,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:42:55,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:42:57,206 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:42:58,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:42:58,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:43:00,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:01,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:05,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:11,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:11,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:13,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:43:13,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:43:17,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:21,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:22,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:27,210 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:43:28,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:28,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:43:28,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:43:31,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:33,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:39,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:41,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:43,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:43:43,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:43:45,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:50,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:51,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:43:56,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:43:57,212 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:43:58,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:43:58,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:44:01,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:02,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:07,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:11,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:13,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:13,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:44:13,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:44:18,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:44:18,809 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:44:18,809 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:44:18,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:18,810 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:44:19,542 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:44:19,543 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:44:21,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:24,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:27,214 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:44:28,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:44:28,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:44:30,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:31,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:36,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:41,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:42,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:43,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:44:43,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:44:48,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:51,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:44:54,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:44:57,220 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:44:58,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:44:58,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:45:00,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:01,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:06,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:11,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:12,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:13,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:45:13,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:45:17,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:21,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:23,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:27,219 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:45:28,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:45:28,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:45:28,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:31,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:34,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:39,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:41,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:43,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:45:43,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:45:45,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:50,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:51,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:45:55,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:45:57,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:45:58,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:45:58,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:46:00,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:01,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:06,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:11,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:12,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:13,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:46:13,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:46:18,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:20,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:46:20,520 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:46:20,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:46:20,521 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:46:20,588 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:46:21,588 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:46:21,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:24,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:27,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:46:28,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:46:28,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:46:29,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:31,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:35,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:41,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:41,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:43,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:46:43,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:46:47,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:51,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:46:53,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:46:57,227 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:46:58,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:46:58,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:46:58,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:01,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:04,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:10,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:11,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:13,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:47:13,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:47:16,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:21,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:21,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:27,229 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:47:27,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:28,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:47:28,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:47:31,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:32,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:38,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:41,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:43,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:43,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:47:43,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:47:48,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:51,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:47:54,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:47:57,231 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:47:58,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:47:58,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:47:59,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:01,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:05,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:11,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:11,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:13,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:48:13,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:48:16,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:21,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:22,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:23,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:48:23,904 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:48:23,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:48:23,905 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:48:24,637 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:48:25,637 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:48:27,234 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:48:28,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:28,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:48:28,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:48:31,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:33,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:38,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:41,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:43,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:48:43,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:48:43,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:49,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:51,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:48:54,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:48:57,237 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:48:58,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:48:58,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:48:59,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:01,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:04,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:10,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:11,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:13,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:49:13,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:49:15,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:20,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:21,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:25,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:27,243 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:49:28,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:49:28,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:49:31,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:31,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:36,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:41,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:41,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:43,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:49:43,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:49:47,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:51,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:49:52,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:57,242 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:49:58,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:49:58,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:49:58,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:50:01,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:03,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:08,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:11,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:13,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:50:13,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:50:13,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:19,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:21,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:24,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:25,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:50:25,729 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:50:25,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:50:25,731 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:50:26,682 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:50:27,244 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:50:27,683 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:50:28,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:50:28,622 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:50:30,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:31,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:35,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:40,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:41,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:43,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:50:43,623 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:50:46,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:51,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:50:52,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:57,246 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:50:58,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:50:58,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:50:58,638 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:51:01,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:03,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:09,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:11,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:13,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:51:13,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:51:15,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:20,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:21,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:25,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:27,248 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:51:28,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:51:28,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:51:31,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:31,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:36,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:41,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:42,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:43,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:51:43,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:51:48,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:51,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:51:53,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:51:57,251 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:51:58,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:51:58,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:51:58,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:01,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:04,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:09,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:11,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:13,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:52:13,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:52:15,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:20,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:21,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:25,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:27,254 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:52:27,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:52:27,386 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:52:27,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:52:27,387 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:52:27,732 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:52:28,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:52:28,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:52:29,733 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:52:30,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:31,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:35,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:41,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:41,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:43,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:52:43,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:52:47,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:51,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:52:53,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:52:57,256 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:52:58,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:52:58,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:52:58,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:01,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:04,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:10,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:11,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:13,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:53:13,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:53:16,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:21,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:21,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:27,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:27,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:53:28,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:53:28,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:53:31,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:33,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:38,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:41,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:43,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:53:43,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:53:43,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:49,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:51,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:53:55,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:53:57,264 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:53:58,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:53:58,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:54:00,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:01,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:06,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:11,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:11,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:13,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:54:13,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:54:16,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:21,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:22,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:27,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:27,263 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:54:28,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:54:28,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:54:30,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:54:30,625 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:54:30,626 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:54:30,628 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:54:30,801 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:54:31,802 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:54:31,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:32,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:38,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:41,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:43,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:43,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:54:43,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:54:49,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:51,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:54:55,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:54:57,265 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:54:58,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:54:58,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:55:00,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:01,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:06,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:11,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:12,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:13,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:55:13,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:55:17,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:21,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:23,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:27,269 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:55:28,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:28,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:55:28,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:55:31,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:34,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:40,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:41,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:43,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:55:43,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:55:46,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:51,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:51,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:55:56,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:55:57,270 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:55:58,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:55:58,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:56:01,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:01,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:07,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:11,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:12,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:13,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:56:13,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:56:18,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:21,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:24,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:27,271 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:56:28,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:56:28,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:56:29,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:31,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:32,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:56:32,279 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:56:32,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:56:32,280 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:56:32,856 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:56:33,858 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:56:35,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:40,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:41,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:43,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:56:43,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:56:46,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:51,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:51,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:56:57,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:56:57,274 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:56:58,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:56:58,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:57:01,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:03,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:08,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:11,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:13,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:57:13,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:57:13,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:19,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:21,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:25,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:27,276 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:57:28,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:57:28,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:57:30,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:31,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:36,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:41,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:42,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:43,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:57:43,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:57:48,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:51,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:57:54,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:57:57,278 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:57:58,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:57:58,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:57:59,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:01,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:05,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:11,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:11,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:13,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:58:13,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:58:17,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:21,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:22,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:27,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:58:27,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:28,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:58:28,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:58:31,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:33,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:33,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 03:58:33,935 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 03:58:33,935 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 03:58:33,936 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 03:58:34,898 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 03:58:35,899 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 03:58:39,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:41,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:43,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:58:43,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:58:44,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:50,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:51,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:58:55,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:58:57,286 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:58:58,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:58:58,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:59:00,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:01,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:06,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:11,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:12,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:13,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:59:13,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:59:18,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:21,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:24,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:27,284 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:59:28,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:59:28,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:59:29,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:31,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:35,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:40,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:41,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:43,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:59:43,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 03:59:46,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:51,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:51,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 03:59:57,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 03:59:57,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 03:59:58,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 03:59:58,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:00:01,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:02,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:08,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:11,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:13,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:00:13,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:00:13,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:19,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:21,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:25,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:27,289 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:00:28,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:00:28,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:00:30,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:31,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:35,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:37,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:00:37,238 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:00:37,238 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:00:37,239 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:00:37,956 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:00:39,957 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:00:40,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:41,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:43,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:00:43,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:00:45,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:51,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:51,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:00:56,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:00:57,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:00:58,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:00:58,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:01:01,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:01,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:06,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:11,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:11,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:13,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:01:13,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:01:17,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:21,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:22,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:27,294 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:01:28,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:28,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:01:28,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:01:31,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:33,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:38,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:41,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:43,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:01:43,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:01:43,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:49,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:51,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:01:54,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:01:57,297 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:01:58,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:01:58,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:01:59,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:01,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:05,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:10,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:11,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:13,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:02:13,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:02:15,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:21,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:21,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:27,299 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:02:27,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:28,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:02:28,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:02:31,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:32,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:37,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:39,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:02:39,005 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:02:39,005 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:02:39,006 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:02:39,008 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:02:40,009 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:02:41,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:43,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:43,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:02:43,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:02:48,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:51,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:02:54,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:02:57,301 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:02:58,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:02:58,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:02:59,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:01,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:05,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:11,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:11,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:13,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:03:13,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:03:16,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:21,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:22,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:27,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:03:28,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:28,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:03:28,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:03:31,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:33,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:39,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:41,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:43,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:03:43,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:03:44,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:50,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:51,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:03:55,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:03:57,306 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:03:58,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:03:58,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:04:00,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:01,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:06,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:11,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:12,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:13,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:04:13,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:04:17,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:21,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:22,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:27,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:04:28,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:28,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:04:28,671 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:04:31,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:33,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:38,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:40,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:04:40,718 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:04:40,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:04:40,718 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:04:41,064 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:04:41,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:42,064 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:04:43,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:04:43,671 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:04:43,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:49,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:51,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:04:54,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:04:57,311 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:04:58,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:04:58,671 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:04:59,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:01,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:05,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:10,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:11,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:13,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:05:13,671 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:05:15,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:21,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:21,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:27,313 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:05:27,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:28,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:05:28,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:05:31,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:32,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:38,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:41,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:43,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:05:43,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:05:43,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:49,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:51,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:05:55,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:05:57,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:05:58,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:05:58,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:06:01,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:01,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:06,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:11,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:12,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:13,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:06:13,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:06:18,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:21,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:24,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:27,317 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:06:28,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:06:28,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:06:30,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:31,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:35,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:41,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:41,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:43,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:06:43,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:06:44,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:06:44,140 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:06:44,140 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:06:44,141 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:06:45,119 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:06:46,120 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:06:46,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:51,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:51,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:06:57,320 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:06:57,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:06:58,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:06:58,673 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:07:01,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:02,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:07,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:11,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:13,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:07:13,673 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:07:13,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:18,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:21,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:23,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:27,322 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:07:28,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:07:28,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:07:29,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:31,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:34,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:39,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:41,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:43,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:07:43,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:07:45,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:50,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:51,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:07:55,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:07:57,328 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:07:58,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:07:58,673 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:08:00,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:01,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:06,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:11,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:12,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:13,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:08:13,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:08:17,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:21,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:23,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:27,326 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:08:28,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:08:28,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:08:29,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:31,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:34,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:40,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:41,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:43,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:08:43,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:08:45,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:08:45,735 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:08:45,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:08:45,737 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:08:45,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:46,178 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:08:48,179 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:08:50,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:51,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:08:55,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:08:57,329 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:08:58,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:08:58,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:09:00,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:01,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:05,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:10,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:11,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:13,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:09:13,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:09:16,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:21,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:21,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:26,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:27,333 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:09:28,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:09:28,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:09:32,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:32,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:37,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:42,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:43,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:09:43,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:09:43,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:48,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:52,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:09:53,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:09:57,334 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:09:58,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:09:58,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:09:59,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:02,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:04,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:09,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:12,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:13,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:10:13,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:10:15,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:20,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:22,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:25,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:27,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:10:28,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:10:28,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:10:31,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:32,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:36,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:42,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:42,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:43,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:10:43,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:10:47,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:47,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:10:47,258 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:10:47,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:10:47,259 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:10:48,223 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:10:48,224 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:10:52,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:52,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:10:57,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:10:57,338 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:10:58,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:10:58,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:11:02,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:03,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:08,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:12,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:13,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:13,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:11:13,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:11:19,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:22,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:25,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:27,340 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:11:28,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:11:28,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:11:30,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:32,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:36,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:41,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:42,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:43,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:11:43,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:11:46,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:52,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:52,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:11:57,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:11:57,342 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:11:58,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:11:58,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:12:02,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:03,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:09,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:12,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:13,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:12:13,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:12:14,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:20,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:22,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:25,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:27,345 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:12:28,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:12:28,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:12:30,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:32,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:36,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:42,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:42,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:43,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:12:43,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:12:48,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:50,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:12:50,372 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:12:50,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:12:50,373 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:12:51,265 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:12:52,266 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:12:52,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:12:54,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:12:57,347 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:12:58,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:12:58,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:12:59,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:02,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:05,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:11,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:12,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:13,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:13:13,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:13:17,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:22,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:23,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:27,350 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:13:28,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:28,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:13:28,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:13:32,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:34,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:40,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:42,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:43,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:13:43,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:13:45,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:51,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:52,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:13:56,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:13:57,352 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:13:58,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:13:58,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:14:01,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:02,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:07,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:12,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:13,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:13,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:14:13,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:14:19,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:22,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:24,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:27,354 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:14:28,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:14:28,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:14:29,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:32,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:35,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:41,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:42,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:43,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:14:43,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:14:46,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:51,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:52,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:14:52,070 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:14:52,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:14:52,071 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:14:52,323 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:14:52,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:14:54,324 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:14:57,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:14:57,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:14:58,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:14:58,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:15:02,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:03,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:09,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:12,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:13,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:15:13,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:15:14,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:20,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:22,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:26,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:27,359 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:15:28,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:15:28,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:15:31,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:32,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:37,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:42,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:43,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:43,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:15:43,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:15:48,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:52,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:15:54,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:15:57,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:15:58,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:15:58,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:15:59,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:02,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:05,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:11,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:12,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:13,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:16:13,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:16:17,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:22,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:23,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:27,364 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:16:28,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:28,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:16:28,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:16:32,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:34,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:39,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:42,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:43,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:16:43,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:16:44,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:50,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:52,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:16:53,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:16:53,958 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:16:53,959 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:16:53,960 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:16:54,372 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:16:56,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:16:56,373 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:16:57,367 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:16:58,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:16:58,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:17:01,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:02,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:07,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:12,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:13,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:13,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:17:13,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:17:19,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:22,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:24,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:27,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:17:28,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:17:28,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:17:29,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:32,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:35,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:40,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:42,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:43,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:17:43,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:17:45,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:51,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:52,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:17:56,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:17:57,371 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:17:58,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:17:58,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:18:01,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:02,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:07,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:12,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:12,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:13,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:18:13,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:18:17,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:22,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:22,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:27,374 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:18:27,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:28,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:18:28,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:18:32,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:33,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:38,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:42,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:43,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:18:43,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:18:43,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:49,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:52,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:18:55,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:18:55,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:18:55,607 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:18:55,607 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:18:55,608 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:18:56,438 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:18:57,376 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:18:58,439 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:18:58,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:18:58,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:19:00,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:02,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:06,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:12,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:12,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:13,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:19:13,685 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:19:17,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:22,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:23,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:27,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:19:28,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:28,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:19:28,685 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:19:32,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:34,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:40,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:42,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:43,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:19:43,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:19:45,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:51,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:52,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:19:57,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:19:57,381 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:19:58,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:19:58,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:20:02,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:03,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:08,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:12,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:13,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:13,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:20:13,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:20:19,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:22,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:25,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:27,384 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:20:28,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:20:28,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:20:30,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:32,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:36,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:41,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:42,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:43,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:20:43,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:20:46,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:52,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:52,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:20:57,386 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:20:57,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:20:58,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:20:58,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:20:58,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:20:58,947 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:20:58,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:20:58,948 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:20:59,493 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:21:00,493 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:21:02,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:03,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:09,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:12,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:13,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:21:13,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:21:14,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:20,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:22,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:25,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:27,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:21:28,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:21:28,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:21:30,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:32,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:36,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:41,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:42,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:43,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:21:43,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:21:47,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:52,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:52,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:21:57,400 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:21:57,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:21:58,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:21:58,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:22:02,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:03,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:08,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:12,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:13,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:22:13,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:22:13,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:19,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:22,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:25,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:27,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:22:28,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:22:28,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:22:30,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:32,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:36,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:41,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:42,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:43,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:22:43,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:22:47,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:52,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:52,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:22:57,397 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:22:57,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:22:58,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:22:58,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:23:00,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:23:00,763 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:23:00,763 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:23:00,764 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:23:01,551 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:23:02,552 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:23:02,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:03,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:08,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:12,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:13,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:23:13,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:23:13,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:19,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:22,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:24,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:27,398 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:23:28,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:23:28,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:23:29,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:32,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:35,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:41,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:42,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:43,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:23:43,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:23:47,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:52,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:23:53,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:57,400 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:23:58,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:23:58,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:23:58,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:24:02,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:04,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:09,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:12,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:13,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:24:13,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:24:14,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:20,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:22,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:27,402 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:24:28,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:24:28,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:24:32,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:32,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:37,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:42,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:42,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:43,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:24:43,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:24:48,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:52,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:24:53,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:57,404 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:24:58,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:24:58,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:24:58,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:25:02,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:25:02,680 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:25:02,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:25:02,682 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:25:02,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:03,610 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:25:03,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:04,611 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:25:09,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:12,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:13,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:25:13,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:25:14,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:20,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:22,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:26,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:27,407 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:25:28,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:25:28,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:25:32,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:33,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:38,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:43,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:25:43,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:25:43,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:43,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:49,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:53,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:25:55,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:25:57,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:25:58,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:25:58,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:26:00,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:03,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:06,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:11,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:13,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:26:13,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:26:13,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:17,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:22,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:23,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:27,417 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:26:27,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:28,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:26:28,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:26:33,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:33,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:39,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:43,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:26:43,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:26:43,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:44,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:50,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:53,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:26:56,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:26:57,414 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:26:58,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:26:58,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:27:01,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:03,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:05,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:27:05,982 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:27:05,982 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:27:05,984 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:27:06,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:06,687 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:27:08,688 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:27:12,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:13,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:27:13,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:27:13,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:18,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:23,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:24,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:27,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:27:28,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:27:28,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:27:29,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:33,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:35,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:40,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:43,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:27:43,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:27:43,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:45,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:51,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:53,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:27:56,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:27:57,419 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:27:58,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:27:58,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:28:02,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:03,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:08,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:13,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:13,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:28:13,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:28:13,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:19,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:23,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:25,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:27,420 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:28:28,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:28:28,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:28:30,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:33,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:36,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:42,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:43,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:28:43,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:28:43,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:47,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:53,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:53,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:28:57,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:28:58,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:28:58,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:28:58,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:29:03,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:04,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:07,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:29:07,753 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:29:07,753 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:29:07,754 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:29:08,746 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:29:08,746 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:29:09,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:13,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:29:13,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:29:13,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:14,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:20,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:23,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:25,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:27,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:29:28,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:29:28,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:29:30,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:33,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:36,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:42,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:43,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:29:43,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:29:43,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:48,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:53,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:53,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:29:57,427 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:29:58,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:29:58,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:29:58,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:30:03,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:04,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:10,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:13,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:30:13,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:30:13,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:15,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:21,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:23,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:26,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:27,429 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:30:28,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:30:28,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:30:32,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:33,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:38,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:43,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:43,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:30:43,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:30:43,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:48,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:53,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:53,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:30:57,436 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:30:58,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:30:58,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:30:58,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:31:03,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:04,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:09,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:31:09,678 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:31:09,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:31:09,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:09,679 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:31:09,804 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:31:10,805 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:31:13,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:31:13,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:31:13,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:14,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:20,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:23,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:25,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:27,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:31:28,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:31:28,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:31:30,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:33,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:36,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:42,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:43,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:31:43,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:31:43,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:48,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:53,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:53,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:31:57,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:31:58,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:31:58,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:31:58,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:32:04,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:04,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:10,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:13,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:32:13,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:32:14,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:15,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:21,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:24,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:26,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:27,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:32:28,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:32:28,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:32:32,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:34,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:37,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:42,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:43,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:32:43,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:32:44,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:48,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:53,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:54,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:32:57,441 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:32:58,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:32:58,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:32:58,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:33:04,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:04,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:09,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:13,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:33:13,010 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:33:13,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:33:13,011 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:33:13,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:33:13,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:33:13,865 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:33:14,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:14,866 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:33:14,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:20,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:24,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:25,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:27,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:33:28,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:33:28,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:33:30,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:34,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:36,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:42,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:43,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:33:43,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:33:44,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:47,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:52,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:54,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:33:57,445 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:33:57,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:33:58,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:33:58,707 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:34:03,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:04,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:08,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:13,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:34:13,707 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:34:13,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:14,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:19,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:24,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:25,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:27,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:34:28,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:34:28,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:34:30,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:34,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:36,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:41,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:43,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:34:43,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:34:44,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:46,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:52,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:54,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:34:57,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:34:57,449 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:34:58,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:34:58,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:35:02,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:04,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:08,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:13,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:35:13,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:35:13,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:14,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:14,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:35:14,905 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:35:14,905 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:35:14,906 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:35:14,928 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:35:16,929 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:35:19,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:24,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:25,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:27,452 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:35:28,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:35:28,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:35:30,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:34,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:36,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:42,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:43,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:35:43,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:35:44,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:48,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:54,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:35:54,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:35:57,457 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:35:58,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:35:58,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:35:59,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:04,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:05,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:11,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:13,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:36:13,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:36:14,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:16,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:21,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:24,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:26,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:27,455 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:36:28,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:36:28,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:36:31,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:34,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:36,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:42,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:43,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:36:43,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:36:44,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:47,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:53,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:54,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:36:57,460 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:36:58,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:36:58,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:36:58,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:37:04,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:04,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:10,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:13,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:37:13,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:37:14,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:15,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:16,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:37:16,634 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:37:16,634 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:37:16,635 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:37:16,985 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:37:18,987 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:37:21,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:24,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:27,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:27,461 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:37:28,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:37:28,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:37:33,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:34,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:39,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:43,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:37:43,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:37:44,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:44,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:50,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:54,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:37:56,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:37:57,463 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:37:58,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:37:58,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:38:01,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:04,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:06,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:12,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:13,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:38:13,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:38:14,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:17,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:22,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:24,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:27,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:27,466 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:38:28,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:38:28,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:38:33,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:34,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:39,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:43,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:38:43,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:38:44,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:44,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:50,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:54,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:38:55,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:38:57,469 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:38:58,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:38:58,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:39:00,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:04,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:06,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:11,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:13,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:39:13,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:39:14,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:17,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:19,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:39:19,860 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:39:19,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:39:19,862 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:39:20,050 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:39:21,050 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:39:22,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:24,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:27,470 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:39:27,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:28,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:39:28,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:39:33,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:34,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:39,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:43,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:39:43,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:39:44,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:44,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:50,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:54,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:39:56,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:39:57,473 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:39:58,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:39:58,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:40:01,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:04,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:06,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:12,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:13,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:40:13,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:40:14,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:18,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:23,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:24,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:27,479 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:40:28,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:28,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:40:28,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:40:34,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:34,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:40,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:43,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:40:43,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:40:44,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:45,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:51,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:54,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:40:57,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:40:57,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:40:58,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:40:58,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:41:03,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:04,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:09,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:13,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:41:13,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:41:14,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:14,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:20,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:21,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:41:21,688 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:41:21,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:41:21,690 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:41:22,108 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:41:23,109 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:41:24,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:25,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:27,480 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:41:28,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:41:28,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:41:30,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:34,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:36,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:42,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:43,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:41:43,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:41:44,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:48,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:53,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:54,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:41:57,485 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:41:58,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:41:58,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:41:58,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:42:04,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:04,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:10,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:13,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:42:13,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:42:14,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:15,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:21,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:24,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:27,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:27,485 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:42:28,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:42:28,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:42:32,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:34,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:38,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:43,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:42:43,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:42:43,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:44,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:49,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:54,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:42:54,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:42:57,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:42:58,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:42:58,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:42:59,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:04,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:05,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:11,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:13,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:43:13,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:43:14,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:17,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:22,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:23,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:43:23,443 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:43:23,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:43:23,444 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:43:24,169 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:43:24,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:25,170 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:43:27,490 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:43:27,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:28,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:43:28,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:43:33,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:34,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:38,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:43,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:43,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:43:43,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:43:44,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:49,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:54,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:43:54,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:43:57,492 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:43:58,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:43:58,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:43:59,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:04,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:05,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:10,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:13,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:44:13,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:44:14,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:15,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:21,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:24,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:27,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:27,495 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:44:28,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:44:28,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:44:33,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:34,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:39,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:43,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:44:43,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:44:44,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:44,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:50,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:54,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:44:56,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:44:57,569 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:44:58,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:44:58,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:45:01,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:04,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:07,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:12,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:13,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:45:13,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:45:14,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:18,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:24,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:24,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:26,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:45:26,837 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:45:26,837 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:45:26,838 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:45:27,232 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:45:27,500 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:45:28,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:45:28,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:45:29,233 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:45:30,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:34,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:35,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:41,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:43,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:45:43,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:45:44,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:47,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:52,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:54,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:45:57,503 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:45:58,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:45:58,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:45:58,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:46:03,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:04,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:08,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:13,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:46:13,750 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:46:14,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:14,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:19,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:24,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:24,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:27,504 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:46:28,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:46:28,753 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:46:30,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:34,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:35,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:40,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:43,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:46:43,753 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:46:44,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:46,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:51,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:54,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:46:56,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:46:57,506 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:46:58,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:46:58,754 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:47:01,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:04,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:06,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:11,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:13,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:47:13,754 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:47:14,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:17,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:23,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:24,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:27,508 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:47:28,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:47:28,524 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:47:28,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:47:28,524 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:47:28,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:47:28,753 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:47:29,291 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:47:29,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:30,291 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:47:34,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:34,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:40,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:43,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:47:43,754 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:47:44,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:46,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:51,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:54,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:47:57,510 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:47:57,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:47:58,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:47:58,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:48:02,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:04,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:07,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:13,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:48:13,754 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:48:13,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:14,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:19,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:24,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:25,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:27,512 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:48:28,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:48:28,754 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:48:30,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:34,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:36,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:41,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:43,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:48:43,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:48:44,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:47,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:52,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:54,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:48:57,514 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:48:58,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:48:58,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:48:58,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:49:03,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:04,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:09,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:13,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:49:13,770 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:49:14,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:14,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:20,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:24,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:25,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:27,522 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:49:28,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:49:28,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:49:30,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:49:30,161 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:49:30,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:49:30,162 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:49:30,340 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:49:30,341 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:49:31,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:34,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:36,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:41,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:43,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:49:43,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:49:44,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:47,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:52,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:54,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:49:57,519 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:49:58,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:49:58,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:49:58,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:50:03,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:04,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:09,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:13,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:50:13,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:50:14,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:14,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:20,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:24,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:26,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:27,521 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:50:28,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:50:28,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:50:32,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:34,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:37,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:43,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:50:43,771 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:50:43,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:44,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:48,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:54,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:50:54,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:50:57,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:50:58,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:50:58,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:51:00,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:04,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:05,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:10,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:13,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:51:13,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:51:14,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:15,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:21,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:24,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:26,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:27,526 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:51:28,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:51:28,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:51:32,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:33,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:51:33,597 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:51:33,597 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:51:33,598 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:51:34,395 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:51:34,396 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:51:34,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:38,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:43,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:43,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:51:43,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:51:44,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:49,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:54,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:51:54,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:51:57,528 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:51:58,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:51:58,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:51:59,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:04,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:05,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:11,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:13,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:52:13,773 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:52:14,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:16,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:22,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:24,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:27,531 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:52:27,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:28,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:52:28,773 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:52:33,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:34,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:39,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:43,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:52:43,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:52:44,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:44,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:50,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:54,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:52:55,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:52:57,533 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:52:58,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:52:58,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:53:00,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:04,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:06,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:12,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:13,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:53:13,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:53:14,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:18,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:23,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:24,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:27,535 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:53:28,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:28,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:53:28,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:53:34,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:34,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:35,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:53:35,349 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:53:35,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:53:35,350 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:53:35,456 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:53:36,457 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:53:39,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:43,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:53:43,776 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:53:44,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:44,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:50,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:54,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:53:55,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:53:57,537 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:53:58,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:53:58,776 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:54:00,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:04,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:06,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:12,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:13,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:54:13,776 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:54:14,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:18,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:24,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:24,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:27,543 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:54:28,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:54:28,776 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:54:29,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:34,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:35,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:40,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:43,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:54:43,777 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:54:44,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:45,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:51,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:54,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:54:56,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:54:57,541 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:54:58,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:54:58,778 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:55:01,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:04,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:07,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:12,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:13,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:55:13,777 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:55:14,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:17,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:23,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:24,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:27,545 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:55:28,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:28,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:55:28,777 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:55:34,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:34,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:37,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:55:37,255 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:55:37,255 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:55:37,255 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:55:37,519 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:55:38,520 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:55:39,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:43,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:55:43,778 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:55:44,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:44,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:50,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:54,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:55:56,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:55:57,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:55:58,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:55:58,779 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:56:01,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:04,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:07,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:13,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:13,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:56:13,778 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:56:14,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:19,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:24,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:24,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:27,549 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:56:28,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:56:28,779 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:56:29,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:34,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:35,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:40,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:43,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:56:43,780 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:56:44,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:45,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:51,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:54,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:56:57,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:56:57,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:56:58,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:56:58,779 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:57:02,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:04,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:08,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:13,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:13,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:57:13,779 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:57:14,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:19,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:24,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:24,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:27,554 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:57:28,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:57:28,780 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:57:29,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:34,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:35,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:39,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:57:39,043 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:57:39,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:57:39,044 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:57:39,572 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:57:40,573 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:57:41,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:43,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:57:43,780 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:57:44,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:46,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:52,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:54,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:57:57,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:57:57,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:58:00,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:58:00,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:58:03,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:04,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:09,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:14,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:15,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:58:15,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:58:15,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:21,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:24,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:27,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:27,558 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:58:30,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:58:30,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:58:32,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:34,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:38,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:43,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:44,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:45,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:58:45,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:58:49,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:54,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:58:55,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:58:57,564 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:59:00,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:59:00,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:59:00,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:04,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:06,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:12,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:14,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:15,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:59:15,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:59:17,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:22,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:24,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:27,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:27,563 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 04:59:30,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:59:30,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:59:33,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:34,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:39,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:42,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 04:59:42,292 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 04:59:42,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 04:59:42,295 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 04:59:42,627 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 04:59:42,628 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 04:59:44,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:44,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:45,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 04:59:45,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 04:59:50,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:54,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 04:59:55,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 04:59:57,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:00:00,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:00:00,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:00:00,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:04,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:06,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:12,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:14,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:15,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:00:15,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:00:18,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:23,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:24,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:27,569 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:00:28,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:30,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:00:30,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:00:34,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:34,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:40,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:44,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:45,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:00:45,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:00:45,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:51,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:54,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:00:57,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:00:57,570 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:01:00,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:01:00,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:01:03,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:04,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:08,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:14,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:14,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:15,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:01:15,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:01:19,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:24,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:25,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:27,574 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:01:30,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:01:30,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:01:30,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:34,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:36,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:42,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:44,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:01:44,029 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:01:44,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:01:44,031 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:01:44,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:44,689 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:01:44,690 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:01:45,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:01:45,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:01:47,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:53,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:01:54,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:01:57,576 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:01:58,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:00,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:02:00,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:02:04,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:04,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:10,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:14,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:15,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:02:15,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:02:15,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:21,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:24,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:27,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:27,577 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:02:30,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:02:30,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:02:33,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:34,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:38,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:43,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:44,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:45,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:02:45,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:02:48,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:54,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:02:54,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:02:57,579 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:02:59,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:00,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:03:00,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:03:04,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:05,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:10,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:14,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:15,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:03:15,414 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:03:15,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:21,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:24,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:27,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:27,590 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:03:30,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:03:30,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:03:32,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:34,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:38,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:43,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:44,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:45,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:03:45,411 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:03:45,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:03:45,784 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:03:45,784 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:03:45,785 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:03:46,746 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:03:46,746 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:03:49,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:54,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:03:55,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:03:57,583 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:04:00,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:04:00,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:04:00,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:04,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:06,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:12,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:14,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:15,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:04:15,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:04:18,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:24,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:24,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:27,587 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:04:29,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:30,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:04:30,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:04:34,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:35,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:40,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:44,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:45,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:04:45,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:04:45,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:51,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:54,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:04:57,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:04:57,588 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:05:00,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:05:00,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:05:02,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:04,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:08,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:13,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:14,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:15,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:05:15,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:05:19,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:24,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:24,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:27,590 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:05:29,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:30,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:05:30,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:05:34,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:35,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:40,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:44,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:45,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:05:45,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:05:45,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:49,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:05:49,031 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:05:49,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:05:49,032 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:05:49,802 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:05:50,803 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:05:51,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:54,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:05:56,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:05:57,592 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:06:00,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:06:00,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:06:01,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:04,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:07,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:12,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:14,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:15,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:06:15,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:06:17,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:23,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:24,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:27,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:06:28,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:30,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:06:30,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:06:34,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:34,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:40,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:44,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:45,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:06:45,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:06:45,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:51,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:54,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:06:56,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:06:57,596 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:07:00,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:07:00,414 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:07:01,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:04,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:07,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:12,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:14,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:15,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:07:15,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:07:18,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:24,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:24,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:27,598 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:07:29,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:30,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:07:30,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:07:34,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:35,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:40,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:44,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:45,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:07:45,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:07:45,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:50,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:07:50,782 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:07:50,783 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:07:50,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:50,784 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:07:50,861 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:07:52,862 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:07:54,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:07:56,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:07:57,605 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:08:00,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:08:00,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:08:01,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:04,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:07,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:12,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:14,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:15,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:08:15,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:08:17,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:23,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:24,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:27,602 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:08:28,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:30,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:08:30,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:08:34,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:34,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:39,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:44,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:45,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:08:45,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:08:45,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:51,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:54,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:08:56,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:08:57,607 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:09:00,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:09:00,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:09:01,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:04,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:07,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:13,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:14,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:15,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:09:15,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:09:18,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:23,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:24,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:27,608 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:09:28,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:30,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:09:30,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:09:34,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:34,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:39,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:44,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:45,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:09:45,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:09:45,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:51,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:52,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:09:52,663 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:09:52,664 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:09:52,665 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:09:52,917 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:09:54,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:09:54,918 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:09:56,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:09:57,610 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:10:00,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:10:00,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:10:02,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:04,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:08,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:14,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:14,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:15,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:10:15,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:10:20,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:24,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:26,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:27,612 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:10:30,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:10:30,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:10:31,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:34,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:37,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:43,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:44,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:45,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:10:45,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:10:48,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:53,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:10:54,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:10:57,614 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:10:58,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:00,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:11:00,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:11:04,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:04,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:10,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:14,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:15,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:11:15,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:11:15,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:21,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:24,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:27,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:27,617 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:11:30,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:11:30,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:11:32,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:34,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:37,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:42,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:44,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:45,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:11:45,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:11:47,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:52,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:54,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:11:55,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:11:55,790 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:11:55,790 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:11:55,791 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:11:55,978 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:11:56,978 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:11:57,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:11:57,619 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:12:00,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:12:00,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:12:02,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:04,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:08,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:14,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:14,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:15,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:12:15,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:12:19,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:24,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:25,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:27,626 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:12:30,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:12:30,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:12:31,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:34,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:36,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:42,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:44,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:45,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:12:45,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:12:47,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:53,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:12:54,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:12:57,624 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:12:58,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:00,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:13:00,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:13:04,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:04,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:09,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:14,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:15,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:13:15,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:13:15,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:21,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:24,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:27,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:27,628 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:13:30,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:13:30,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:13:33,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:34,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:38,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:43,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:44,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:45,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:13:45,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:13:49,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:54,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:13:55,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:13:57,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:13:57,344 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:13:57,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:13:57,346 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:13:57,628 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:13:58,031 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:13:59,032 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:14:00,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:14:00,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:14:00,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:04,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:06,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:11,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:14,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:15,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:14:15,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:14:16,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:22,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:24,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:27,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:27,631 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:14:30,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:14:30,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:14:33,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:34,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:38,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:44,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:44,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:45,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:14:45,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:14:49,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:54,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:14:55,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:14:57,633 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:15:00,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:15:00,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:15:00,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:04,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:06,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:12,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:14,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:15,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:15:15,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:15:17,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:22,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:24,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:27,635 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:15:27,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:30,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:15:30,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:15:33,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:34,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:38,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:44,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:44,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:45,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:15:45,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:15:50,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:54,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:15:55,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:15:57,638 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:15:59,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:15:59,024 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:15:59,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:15:59,025 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:15:59,091 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:16:00,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:16:00,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:16:01,092 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:16:01,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:04,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:06,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:12,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:14,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:15,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:16:15,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:16:18,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:23,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:24,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:27,640 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:16:28,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:30,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:16:30,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:16:34,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:34,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:39,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:44,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:45,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:16:45,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:16:45,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:51,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:54,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:16:57,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:16:57,642 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:17:00,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:17:00,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:17:02,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:04,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:07,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:13,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:14,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:15,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:17:15,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:17:18,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:23,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:24,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:27,649 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:17:28,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:30,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:17:30,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:17:34,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:34,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:39,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:44,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:45,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:45,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:17:45,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:17:50,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:54,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:17:55,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:17:57,648 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:18:00,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:00,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:18:00,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:18:02,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:18:02,380 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:18:02,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:18:02,384 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:18:03,151 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:18:03,151 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:18:04,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:06,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:12,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:14,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:15,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:18:15,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:18:17,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:23,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:24,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:27,650 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:18:28,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:30,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:18:30,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:18:34,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:34,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:40,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:44,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:45,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:45,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:18:45,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:18:51,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:54,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:18:57,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:18:57,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:19:00,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:19:00,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:19:02,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:04,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:08,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:13,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:14,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:15,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:19:15,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:19:18,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:24,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:24,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:27,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:19:29,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:30,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:19:30,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:19:34,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:35,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:41,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:44,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:45,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:19:45,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:19:46,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:52,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:54,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:19:57,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:19:57,657 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:20:00,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:20:00,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:20:02,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:04,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:04,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:20:04,557 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:20:04,558 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:20:04,558 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:20:05,211 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:20:05,211 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:20:08,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:14,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:14,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:15,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:20:15,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:20:19,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:24,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:25,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:27,660 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:20:30,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:20:30,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:20:30,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:34,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:36,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:41,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:44,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:45,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:20:45,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:20:46,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:52,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:54,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:20:57,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:20:57,662 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:21:00,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:21:00,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:21:02,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:04,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:08,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:14,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:14,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:15,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:21:15,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:21:20,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:24,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:26,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:27,665 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:21:30,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:21:30,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:21:31,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:34,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:37,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:43,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:44,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:45,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:21:45,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:21:48,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:54,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:21:54,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:21:57,671 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:22:00,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:00,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:22:00,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:22:04,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:06,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:06,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:22:06,771 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:22:06,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:22:06,772 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:22:07,270 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:22:09,271 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:22:12,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:14,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:15,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:22:15,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:22:17,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:23,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:24,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:27,670 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:22:28,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:30,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:22:30,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:22:34,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:34,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:40,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:44,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:45,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:45,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:22:45,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:22:50,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:54,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:22:55,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:22:57,676 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:23:00,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:23:00,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:23:00,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:04,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:06,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:12,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:14,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:15,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:23:15,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:23:17,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:23,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:24,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:27,676 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:23:28,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:30,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:23:30,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:23:34,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:34,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:39,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:44,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:44,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:45,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:23:45,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:23:49,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:54,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:23:55,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:23:57,679 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:23:59,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:00,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:24:00,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:24:05,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:05,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:10,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:10,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:24:10,798 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:24:10,799 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:24:10,799 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:24:11,336 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:24:13,337 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:24:15,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:24:15,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:24:15,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:15,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:21,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:25,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:26,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:27,682 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:24:30,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:24:30,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:24:31,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:35,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:37,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:43,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:45,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:24:45,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:24:45,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:48,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:54,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:24:55,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:24:57,686 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:24:59,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:00,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:25:00,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:25:05,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:05,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:11,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:15,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:25:15,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:25:15,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:16,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:22,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:25,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:27,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:27,689 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:25:30,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:25:30,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:25:32,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:35,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:38,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:43,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:45,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:25:45,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:25:45,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:49,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:54,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:25:55,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:25:57,692 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:25:59,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:00,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:26:00,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:26:05,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:05,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:11,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:12,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:26:12,757 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:26:12,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:26:12,758 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:26:13,387 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:26:13,388 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:26:15,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:26:15,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:26:15,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:16,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:22,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:25,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:27,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:27,698 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:26:30,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:26:30,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:26:33,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:35,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:38,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:43,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:45,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:26:45,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:26:45,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:48,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:54,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:26:55,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:26:57,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:26:59,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:00,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:27:00,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:27:05,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:05,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:11,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:15,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:27:15,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:27:15,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:16,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:22,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:25,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:27,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:27,701 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:27:30,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:27:30,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:27:33,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:35,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:38,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:44,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:45,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:27:45,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:27:45,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:49,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:55,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:27:55,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:27:57,702 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:28:00,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:28:00,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:28:00,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:05,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:06,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:11,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:14,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:28:14,421 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:28:14,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:28:14,423 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:28:14,442 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:28:15,442 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:28:15,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:28:15,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:28:15,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:16,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:22,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:25,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:27,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:27,766 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:28:30,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:28:30,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:28:33,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:35,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:38,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:44,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:45,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:28:45,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:28:45,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:50,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:55,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:28:55,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:28:57,768 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:29:00,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:29:00,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:29:05,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:05,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:10,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:15,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:29:15,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:29:15,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:15,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:21,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:25,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:27,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:27,771 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:29:30,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:29:30,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:29:32,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:35,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:38,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:44,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:45,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:29:45,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:29:45,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:49,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:55,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:29:55,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:29:57,773 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:30:00,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:30:00,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:30:00,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:05,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:06,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:12,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:15,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:30:15,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:30:15,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:16,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:30:16,267 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:30:16,267 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:30:16,268 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:30:16,495 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:30:17,495 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:30:18,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:23,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:25,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:27,776 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:30:28,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:30,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:30:30,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:30:34,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:35,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:40,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:45,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:30:45,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:30:45,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:45,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:51,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:55,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:30:56,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:30:57,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:31:00,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:31:00,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:31:01,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:05,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:07,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:13,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:15,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:31:15,466 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:31:15,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:18,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:24,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:25,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:27,780 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:31:29,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:30,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:31:30,466 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:31:35,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:35,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:41,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:45,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:31:45,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:31:45,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:46,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:52,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:31:55,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:31:57,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:31:57,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:00,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:32:00,467 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:32:03,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:05,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:09,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:14,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:15,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:32:15,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:32:15,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:19,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:32:19,575 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:32:19,576 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:32:19,577 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:32:19,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:20,558 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:32:21,559 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:32:25,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:25,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:27,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:32:30,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:32:30,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:32:30,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:35,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:36,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:42,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:45,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:32:45,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:32:45,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:47,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:53,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:32:55,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:32:57,787 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:32:58,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:00,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:33:00,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:33:04,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:05,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:10,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:15,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:33:15,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:33:15,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:15,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:21,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:25,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:27,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:27,790 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:33:30,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:33:30,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:33:32,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:35,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:38,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:43,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:45,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:33:45,468 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:33:45,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:48,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:54,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:33:55,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:33:57,792 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:33:59,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:00,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:34:00,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:34:05,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:05,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:11,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:15,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:34:15,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:34:15,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:16,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:21,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:34:21,086 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:34:21,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:34:21,087 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:34:21,604 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:34:22,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:23,605 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:34:25,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:27,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:27,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:34:30,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:34:30,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:34:32,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:35,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:38,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:44,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:45,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:34:45,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:34:45,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:49,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:55,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:34:55,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:34:57,798 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:35:00,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:35:00,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:35:01,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:05,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:06,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:11,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:15,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:35:15,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:35:15,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:17,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:22,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:25,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:27,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:27,894 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:35:30,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:35:30,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:35:32,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:35,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:38,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:44,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:45,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:35:45,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:35:45,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:49,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:55,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:35:55,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:35:57,803 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:36:00,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:36:00,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:36:00,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:05,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:06,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:11,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:15,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:36:15,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:36:15,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:16,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:22,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:22,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:36:22,736 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:36:22,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:36:22,739 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:36:23,667 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:36:25,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:25,668 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:36:27,806 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:36:27,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:30,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:36:30,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:36:33,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:35,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:39,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:45,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:36:45,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:36:45,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:45,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:50,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:55,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:36:55,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:36:57,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:37:00,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:37:00,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:37:01,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:05,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:06,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:11,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:15,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:37:15,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:37:15,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:17,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:23,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:25,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:27,811 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:37:29,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:30,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:37:30,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:37:34,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:35,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:40,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:45,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:37:45,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:37:45,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:46,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:51,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:37:55,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:37:57,814 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:37:57,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:00,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:38:00,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:38:02,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:05,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:08,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:13,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:15,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:38:15,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:38:15,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:19,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:25,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:25,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:26,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:38:26,263 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:38:26,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:38:26,264 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:38:26,733 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:38:27,733 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:38:27,816 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:38:30,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:38:30,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:38:31,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:35,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:36,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:42,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:45,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:38:45,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:38:45,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:48,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:53,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:38:55,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:38:57,818 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:38:59,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:00,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:39:00,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:39:04,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:05,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:10,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:15,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:39:15,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:39:15,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:16,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:21,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:25,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:27,820 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:39:27,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:30,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:39:30,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:39:32,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:35,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:37,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:43,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:45,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:39:45,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:39:45,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:49,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:54,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:39:55,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:39:57,826 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:40:00,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:40:00,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:40:00,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:05,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:05,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:11,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:15,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:40:15,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:40:15,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:16,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:22,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:25,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:27,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:40:27,646 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:40:27,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:40:27,647 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:40:27,794 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:40:27,824 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:40:28,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:29,795 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:40:30,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:40:30,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:40:33,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:35,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:38,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:44,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:45,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:40:45,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:40:45,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:50,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:55,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:40:55,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:40:57,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:41:00,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:41:00,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:41:01,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:05,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:06,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:12,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:15,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:41:15,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:41:15,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:18,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:23,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:25,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:27,830 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:41:29,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:30,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:41:30,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:41:34,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:35,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:39,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:45,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:41:45,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:41:45,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:45,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:50,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:55,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:41:56,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:41:57,832 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:42:00,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:42:00,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:42:02,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:05,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:08,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:13,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:15,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:42:15,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:42:15,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:18,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:24,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:25,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:27,833 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:42:29,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:42:29,049 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:42:29,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:42:29,050 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:42:29,844 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:42:30,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:30,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:42:30,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:42:31,844 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:42:35,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:35,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:41,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:45,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:42:45,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:42:45,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:47,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:53,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:42:55,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:42:57,835 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:42:59,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:00,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:43:00,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:43:04,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:05,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:10,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:15,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:43:15,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:43:15,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:16,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:22,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:25,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:27,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:27,837 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:43:30,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:43:30,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:43:33,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:35,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:39,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:45,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:45,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:43:45,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:43:45,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:50,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:55,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:43:56,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:43:57,840 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:44:00,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:44:00,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:44:01,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:05,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:07,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:12,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:15,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:44:15,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:44:15,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:18,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:23,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:25,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:27,846 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:44:28,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:30,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:44:30,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:44:31,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:44:31,952 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:44:31,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:44:31,954 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:44:32,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:44:33,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:44:34,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:35,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:40,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:45,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:44:45,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:44:45,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:45,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:51,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:55,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:44:57,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:44:57,845 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:45:00,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:45:00,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:45:03,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:05,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:08,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:13,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:15,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:45:15,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:45:15,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:19,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:25,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:25,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:27,848 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:45:30,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:45:30,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:45:30,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:35,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:36,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:42,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:45,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:45:45,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:45:45,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:48,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:53,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:45:55,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:45:57,849 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:45:58,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:00,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:46:00,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:46:04,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:05,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:10,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:15,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:15,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:46:15,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:46:15,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:20,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:25,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:25,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:27,851 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:46:30,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:46:30,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:46:30,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:33,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:46:33,710 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:46:33,710 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:46:33,711 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:46:33,952 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:46:35,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:35,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:35,953 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:46:41,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:45,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:46:45,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:46:45,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:46,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:52,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:46:55,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:46:57,853 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:46:57,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:00,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:47:00,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:47:03,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:05,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:09,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:14,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:15,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:47:15,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:47:15,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:19,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:24,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:25,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:27,855 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:47:29,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:30,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:47:30,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:47:35,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:35,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:41,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:45,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:47:45,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:47:45,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:46,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:52,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:47:55,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:47:57,857 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:47:57,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:00,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:48:00,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:48:03,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:05,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:08,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:14,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:15,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:48:15,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:48:15,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:19,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:25,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:25,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:27,860 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:48:30,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:48:30,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:48:30,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:35,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:48:35,397 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:48:35,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:48:35,398 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:48:35,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:36,003 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:48:36,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:38,004 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:48:42,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:45,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:48:45,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:48:45,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:48,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:53,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:48:55,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:48:57,862 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:48:58,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:00,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:49:00,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:49:04,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:05,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:09,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:15,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:15,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:49:15,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:49:15,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:21,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:25,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:27,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:27,869 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:49:30,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:49:30,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:49:33,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:35,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:38,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:44,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:45,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:49:45,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:49:45,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:50,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:55,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:49:55,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:49:57,865 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:50:00,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:50:00,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:50:00,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:05,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:06,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:12,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:15,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:50:15,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:50:15,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:17,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:22,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:25,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:27,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:27,869 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:50:30,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:50:30,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:50:32,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:35,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:38,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:38,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:50:38,559 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:50:38,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:50:38,560 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:50:39,050 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:50:40,051 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:50:43,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:45,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:50:45,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:50:45,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:49,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:54,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:50:55,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:50:57,871 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:50:59,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:00,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:51:00,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:51:05,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:05,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:11,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:15,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:51:15,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:51:15,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:16,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:22,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:25,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:27,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:51:27,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:30,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:51:30,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:51:33,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:35,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:38,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:44,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:45,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:51:45,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:51:45,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:49,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:55,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:51:55,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:51:57,875 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:52:00,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:52:00,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:52:00,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:05,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:06,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:12,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:15,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:52:15,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:52:15,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:18,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:24,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:25,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:27,877 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:52:29,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:30,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:52:30,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:52:35,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:35,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:40,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:52:40,278 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:52:40,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:52:40,279 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:52:41,100 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:52:41,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:42,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:52:45,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:52:45,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:52:45,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:46,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:52,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:55,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:52:57,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:52:57,879 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:53:00,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:53:00,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:53:03,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:05,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:08,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:13,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:15,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:53:15,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:53:15,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:19,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:24,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:25,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:27,882 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:53:29,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:30,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:53:30,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:53:35,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:35,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:40,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:45,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:45,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:53:45,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:53:45,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:51,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:55,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:53:57,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:53:57,887 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:54:00,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:54:00,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:54:02,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:05,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:08,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:14,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:15,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:54:15,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:54:15,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:20,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:25,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:25,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:27,885 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:54:30,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:54:30,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:54:30,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:35,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:36,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:41,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:41,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:54:41,934 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:54:41,934 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:54:41,936 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:54:42,149 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:54:44,150 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:54:45,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:54:45,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:54:45,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:46,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:52,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:55,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:54:57,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:54:57,888 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:55:00,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:55:00,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:55:02,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:05,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:07,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:13,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:15,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:55:15,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:55:15,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:19,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:24,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:25,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:27,891 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:55:29,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:30,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:55:30,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:55:35,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:35,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:40,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:45,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:55:45,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:55:45,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:45,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:51,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:55,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:55:57,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:55:57,892 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:56:00,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:56:00,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:56:02,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:07,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:08,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:14,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:15,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:56:15,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:56:17,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:19,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:25,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:27,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:27,894 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:56:30,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:56:30,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:56:30,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:36,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:37,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:42,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:45,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:56:45,157 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:56:45,157 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:56:45,158 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:56:45,207 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:56:45,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:56:45,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:56:47,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:48,208 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:56:48,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:53,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:56:57,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:56:57,896 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:56:58,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:00,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:57:00,511 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:57:04,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:07,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:10,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:15,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:15,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:57:15,510 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:57:17,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:20,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:25,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:27,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:27,898 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:57:30,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:57:30,511 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:57:30,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:36,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:37,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:42,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:45,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:57:45,511 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:57:47,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:48,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:53,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:57:57,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:57:57,900 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:57:58,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:00,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:58:00,512 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:58:04,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:07,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:10,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:15,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:58:15,512 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:58:15,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:17,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:21,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:27,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:27,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:27,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:58:30,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:58:30,512 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:58:33,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:37,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:39,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:44,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:45,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:58:45,513 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:58:46,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 05:58:46,801 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 05:58:46,801 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 05:58:46,802 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 05:58:47,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:47,252 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 05:58:48,252 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 05:58:49,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:55,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:58:57,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:58:57,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:59:00,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:59:00,512 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:59:00,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:06,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:07,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:12,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:15,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:59:15,512 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:59:17,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:18,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:23,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:27,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:27,909 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 05:59:28,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:30,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:59:30,513 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:59:34,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:37,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:40,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:45,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:45,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 05:59:45,513 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 05:59:47,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:51,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:56,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 05:59:57,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 05:59:57,910 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:00:00,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:00:00,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:00:01,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:07,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:07,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:12,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:15,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:00:15,513 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:00:17,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:18,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:23,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:27,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:27,911 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:00:28,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:30,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:00:30,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:00:34,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:37,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:39,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:44,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:45,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:00:45,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:00:47,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:48,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:00:48,507 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:00:48,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:00:48,508 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:00:49,298 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:00:49,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:50,298 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:00:55,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:00:57,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:00:57,914 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:01:00,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:01:00,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:01:00,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:06,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:07,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:12,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:15,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:01:15,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:01:17,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:17,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:22,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:27,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:27,917 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:01:27,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:30,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:01:30,515 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:01:33,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:37,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:39,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:44,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:45,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:01:45,515 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:01:47,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:50,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:55,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:01:57,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:01:57,919 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:02:00,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:02:00,515 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:02:00,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:06,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:07,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:12,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:15,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:02:15,516 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:02:17,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:17,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:23,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:27,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:27,921 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:02:28,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:31,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:02:31,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:02:34,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:37,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:40,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:46,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:46,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:02:46,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:02:47,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:52,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:02:52,058 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:02:52,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:02:52,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:52,060 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:02:52,361 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:02:54,362 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:02:57,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:02:57,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:02:57,926 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:03:01,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:03:01,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:03:02,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:07,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:08,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:14,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:16,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:03:16,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:03:17,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:19,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:24,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:27,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:27,926 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:03:29,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:31,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:03:31,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:03:35,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:37,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:40,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:45,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:46,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:03:46,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:03:47,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:51,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:56,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:03:57,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:03:57,930 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:04:01,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:01,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:04:01,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:04:07,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:07,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:12,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:16,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:04:16,999 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:04:17,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:17,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:22,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:27,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:27,930 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:04:27,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:31,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:04:31,999 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:04:33,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:37,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:38,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:44,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:46,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:04:46,999 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:04:47,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:50,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:53,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:04:53,691 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:04:53,692 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:04:53,692 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:04:54,414 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:04:55,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:04:56,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:04:57,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:04:57,933 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:05:00,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:01,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:05:02,000 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:05:06,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:07,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:11,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:16,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:05:17,000 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:05:17,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:17,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:22,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:27,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:27,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:27,936 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:05:32,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:05:32,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:05:33,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:37,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:38,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:43,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:47,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:05:47,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:05:47,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:49,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:54,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:05:57,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:05:57,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:05:59,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:02,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:06:02,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:06:05,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:07,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:11,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:17,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:06:17,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:06:17,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:17,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:22,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:27,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:27,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:27,940 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:06:32,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:06:32,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:06:33,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:37,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:38,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:44,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:47,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:06:47,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:06:47,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:50,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:55,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:06:55,308 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:06:55,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:06:55,310 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:06:55,464 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:06:56,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:06:57,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:06:57,942 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:06:58,465 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:07:01,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:02,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:07:02,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:07:07,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:07,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:12,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:17,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:07:17,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:07:17,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:18,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:23,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:27,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:27,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:07:28,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:32,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:07:32,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:07:34,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:37,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:40,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:46,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:47,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:07:47,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:07:47,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:51,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:57,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:07:57,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:07:57,952 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:08:02,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:08:02,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:08:03,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:07,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:09,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:14,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:17,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:08:17,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:08:17,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:20,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:25,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:27,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:27,948 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:08:30,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:32,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:08:32,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:08:36,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:37,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:41,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:47,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:08:47,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:08:47,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:47,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:52,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:56,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:08:56,924 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:08:56,924 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:08:56,926 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:08:57,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:08:57,524 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:08:57,950 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:08:57,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:08:58,524 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:09:02,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:09:02,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:09:03,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:07,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:09,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:14,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:17,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:09:17,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:09:17,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:20,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:25,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:27,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:27,954 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:09:30,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:32,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:09:32,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:09:36,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:37,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:42,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:47,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:09:47,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:09:47,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:48,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:53,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:09:57,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:09:57,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:09:58,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:02,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:10:02,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:10:04,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:07,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:10,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:16,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:17,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:10:17,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:10:17,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:22,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:27,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:27,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:27,958 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:10:32,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:10:32,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:10:33,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:37,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:38,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:43,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:47,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:10:47,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:10:47,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:49,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:55,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:10:57,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:10:57,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:11:00,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:11:00,111 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:11:00,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:11:00,112 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:11:00,582 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:11:01,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:02,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:11:02,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:11:02,583 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:11:06,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:07,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:12,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:17,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:11:17,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:11:17,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:17,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:22,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:27,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:27,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:27,962 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:11:32,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:11:32,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:11:32,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:37,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:38,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:44,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:47,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:11:47,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:11:47,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:49,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:54,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:11:57,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:11:57,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:11:59,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:02,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:12:02,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:12:05,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:07,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:11,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:16,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:17,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:12:17,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:12:17,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:22,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:27,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:27,970 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:12:27,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:32,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:12:32,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:12:33,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:37,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:38,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:44,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:47,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:12:47,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:12:47,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:50,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:55,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:12:57,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:12:57,969 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:13:00,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:01,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:13:01,930 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:13:01,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:13:01,932 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:13:02,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:13:02,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:13:02,633 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:13:03,634 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:13:06,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:07,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:11,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:17,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:13:17,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:13:17,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:17,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:22,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:27,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:27,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:27,971 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:13:32,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:13:32,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:13:32,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:37,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:37,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:43,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:47,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:13:47,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:13:47,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:48,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:54,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:13:57,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:13:57,975 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:13:59,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:02,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:14:02,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:14:05,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:07,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:11,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:17,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:14:17,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:14:17,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:17,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:22,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:27,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:27,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:27,976 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:14:32,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:14:32,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:14:32,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:37,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:37,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:43,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:47,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:14:47,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:14:47,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:49,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:55,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:14:57,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:14:57,978 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:15:00,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:02,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:15:02,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:15:03,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:15:03,412 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:15:03,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:15:03,413 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:15:03,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:15:05,702 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:15:06,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:07,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:11,706 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:15:12,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:17,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:15:17,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:15:17,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:17,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:21,713 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:15:23,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:27,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:27,980 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:15:28,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:32,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:15:32,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:15:34,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:37,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:40,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:45,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:47,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:15:47,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:15:47,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:50,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:56,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:15:57,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:15:57,981 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:16:01,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:02,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:16:02,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:16:07,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:07,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:13,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:17,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:16:17,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:16:17,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:18,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:24,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:27,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:27,983 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:16:29,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:32,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:16:32,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:16:35,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:37,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:41,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:46,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:47,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:16:47,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:16:47,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:52,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:16:57,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:16:57,986 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:16:57,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:02,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:17:02,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:17:03,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:07,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:09,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:14,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:17,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:17:17,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:17:17,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:19,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:23,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:17:23,179 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:17:23,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:17:23,180 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:17:23,777 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:17:23,778 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:17:24,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:27,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:27,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:17:29,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:32,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:17:32,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:17:35,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:37,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:41,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:47,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:17:47,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:17:47,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:47,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:53,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:17:57,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:17:57,991 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:17:58,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:02,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:18:02,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:18:04,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:07,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:10,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:16,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:17,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:18:17,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:18:17,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:21,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:27,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:27,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:27,994 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:18:32,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:18:32,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:18:32,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:37,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:38,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:44,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:47,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:18:47,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:18:47,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:49,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:55,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:18:57,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:18:57,997 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:19:00,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:02,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:19:02,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:19:06,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:07,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:11,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:16,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:17,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:19:17,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:19:17,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:22,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:24,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:19:24,647 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:19:24,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:19:24,648 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:19:24,818 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:19:25,819 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:19:27,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:27,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:27,998 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:19:32,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:19:32,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:19:32,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:37,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:38,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:44,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:47,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:19:47,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:19:47,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:49,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:55,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:19:57,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:19:58,001 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:20:01,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:02,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:20:02,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:20:06,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:07,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:12,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:17,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:20:17,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:20:17,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:17,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:23,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:27,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:28,003 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:20:29,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:32,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:20:32,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:20:34,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:37,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:40,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:46,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:47,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:20:47,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:20:47,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:51,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:56,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:20:57,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:20:58,005 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:21:01,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:02,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:21:02,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:21:07,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:07,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:12,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:17,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:21:17,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:21:17,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:17,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:23,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:26,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:21:26,401 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:21:26,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:21:26,402 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:21:26,884 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:21:27,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:27,885 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:21:28,008 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:21:29,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:32,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:21:32,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:21:34,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:37,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:40,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:46,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:47,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:21:47,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:21:47,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:51,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:57,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:21:57,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:21:58,015 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:22:02,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:22:02,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:22:02,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:07,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:08,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:14,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:17,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:22:17,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:22:17,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:19,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:25,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:27,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:28,013 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:22:31,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:32,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:22:32,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:22:36,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:37,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:41,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:46,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:47,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:22:47,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:22:47,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:51,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:57,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:22:57,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:22:58,016 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:23:02,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:23:02,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:23:02,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:07,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:08,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:13,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:17,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:23:17,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:23:17,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:18,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:24,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:27,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:28,017 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:23:29,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:23:29,412 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:23:29,412 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:23:29,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:29,413 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:23:29,945 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:23:31,947 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:23:32,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:23:32,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:23:35,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:37,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:41,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:46,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:47,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:23:47,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:23:47,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:52,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:57,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:23:57,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:23:58,019 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:24:02,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:24:02,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:24:02,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:07,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:08,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:14,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:17,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:24:17,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:24:17,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:19,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:25,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:27,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:28,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:24:31,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:32,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:24:32,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:24:36,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:37,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:41,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:46,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:47,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:24:47,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:24:47,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:52,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:57,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:24:57,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:24:58,025 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:25:02,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:25:02,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:25:02,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:07,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:08,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:14,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:17,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:25:17,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:25:17,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:19,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:25,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:27,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:28,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:25:31,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:31,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:25:31,277 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:25:31,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:25:31,278 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:25:31,996 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:25:31,996 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:25:32,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:25:32,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:25:36,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:37,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:41,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:46,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:47,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:25:47,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:25:47,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:52,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:25:57,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:25:58,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:25:58,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:02,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:26:02,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:26:03,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:07,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:09,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:15,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:17,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:26:17,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:26:17,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:20,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:26,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:27,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:28,036 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:26:32,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:26:32,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:26:32,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:37,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:38,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:43,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:47,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:26:47,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:26:47,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:48,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:54,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:26:57,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:26:58,034 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:27:00,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:02,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:27:02,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:27:05,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:07,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:11,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:17,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:27:17,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:27:17,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:17,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:23,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:27,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:28,038 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:27:29,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:32,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:27:32,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:27:32,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:27:32,810 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:27:32,811 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:27:32,812 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:27:33,043 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:27:34,043 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:27:34,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:37,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:40,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:45,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:47,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:27:47,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:27:47,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:50,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:56,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:27:57,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:27:58,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:28:01,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:02,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:28:02,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:28:07,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:07,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:13,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:17,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:28:17,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:28:17,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:18,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:24,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:27,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:28,042 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:28:30,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:32,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:28:32,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:28:35,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:37,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:41,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:46,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:47,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:28:47,626 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:28:47,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:51,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:57,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:28:57,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:28:58,044 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:29:02,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:29:02,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:29:02,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:07,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:08,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:13,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:17,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:29:17,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:29:17,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:18,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:24,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:27,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:28,046 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:29:30,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:32,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:29:32,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:29:35,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:36,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:29:36,010 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:29:36,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:29:36,011 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:29:36,096 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:29:37,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:38,097 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:29:41,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:46,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:47,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:29:47,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:29:47,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:51,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:57,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:29:57,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:29:58,049 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:30:02,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:30:02,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:30:02,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:07,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:08,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:14,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:17,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:30:17,616 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:30:17,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:19,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:25,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:27,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:28,051 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:30:31,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:32,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:30:32,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:30:36,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:37,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:42,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:47,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:30:47,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:30:47,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:47,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:53,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:30:57,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:30:58,057 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:30:59,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:02,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:31:02,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:31:04,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:07,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:10,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:16,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:17,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:31:17,617 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:31:17,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:21,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:27,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:27,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:28,056 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:31:32,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:31:32,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:31:32,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:37,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:37,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:31:37,751 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:31:37,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:31:37,754 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:31:38,188 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:31:38,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:40,189 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:31:44,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:47,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:31:47,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:31:47,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:49,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:55,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:31:57,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:31:58,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:32:01,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:02,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:32:02,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:32:06,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:07,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:12,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:17,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:32:17,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:32:17,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:17,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:23,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:27,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:28,062 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:32:29,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:32,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:32:32,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:32:34,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:37,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:40,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:45,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:47,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:32:47,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:32:47,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:50,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:56,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:32:57,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:32:58,063 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:33:02,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:02,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:33:02,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:33:07,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:07,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:12,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:17,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:33:17,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:33:17,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:18,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:23,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:27,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:28,066 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:33:29,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:32,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:33:32,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:33:34,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:37,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:39,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:33:39,479 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:33:39,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:33:39,480 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:33:40,242 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:33:40,243 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:33:40,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:45,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:47,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:33:47,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:33:47,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:50,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:56,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:33:57,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:33:58,068 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:34:02,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:02,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:34:02,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:34:07,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:07,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:13,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:17,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:34:17,620 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:34:17,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:18,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:24,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:27,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:28,071 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:34:30,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:32,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:34:32,622 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:34:35,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:37,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:40,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:46,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:47,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:34:47,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:34:47,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:51,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:57,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:34:57,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:34:58,073 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:35:02,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:35:02,621 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:35:02,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:07,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:08,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:13,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:17,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:17,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:35:17,843 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:35:18,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:24,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:27,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:28,075 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:35:30,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:32,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:35:32,844 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:35:36,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:37,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:41,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:42,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:35:42,744 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:35:42,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:35:42,745 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:35:43,292 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:35:44,293 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:35:46,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:47,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:47,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:35:47,843 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:35:52,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:57,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:35:57,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:35:58,078 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:36:02,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:36:02,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:36:03,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:07,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:08,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:13,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:17,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:17,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:36:17,844 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:36:19,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:24,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:27,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:28,080 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:36:30,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:32,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:36:32,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:36:35,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:37,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:41,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:46,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:47,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:47,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:36:47,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:36:52,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:57,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:36:57,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:36:58,082 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:37:02,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:37:02,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:37:02,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:07,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:08,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:13,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:17,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:17,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:37:17,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:37:18,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:24,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:27,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:28,084 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:37:30,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:32,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:37:32,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:37:35,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:37,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:41,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:44,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:37:44,473 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:37:44,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:37:44,475 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:37:45,349 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:37:46,350 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:37:47,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:47,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:47,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:37:47,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:37:52,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:57,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:37:57,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:37:58,088 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:38:02,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:38:02,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:38:03,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:07,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:08,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:13,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:17,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:17,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:38:17,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:38:19,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:24,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:27,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:28,091 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:38:30,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:32,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:38:32,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:38:35,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:37,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:40,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:45,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:47,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:47,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:38:47,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:38:51,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:56,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:38:57,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:38:58,092 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:39:02,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:02,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:39:02,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:39:07,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:07,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:13,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:17,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:17,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:39:17,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:39:19,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:24,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:27,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:28,096 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:39:30,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:32,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:39:32,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:39:36,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:37,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:41,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:46,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:39:46,680 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:39:46,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:39:46,681 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:39:47,411 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:39:47,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:47,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:47,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:39:47,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:39:48,412 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:39:52,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:57,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:39:57,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:39:58,098 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:40:02,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:40:02,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:40:02,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:07,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:08,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:13,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:17,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:17,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:40:17,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:40:19,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:24,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:27,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:28,105 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:40:30,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:32,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:40:32,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:40:36,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:37,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:41,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:46,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:47,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:47,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:40:47,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:40:52,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:57,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:40:57,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:40:58,103 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:41:02,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:41:02,868 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:41:03,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:07,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:08,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:14,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:17,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:17,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:41:17,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:41:20,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:25,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:27,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:28,107 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:41:31,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:32,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:41:32,869 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:41:37,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:37,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:42,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:47,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:49,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:49,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:41:49,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:41:50,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:41:50,232 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:41:50,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:41:50,234 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:41:50,467 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:41:52,468 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:41:53,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:41:58,108 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:41:59,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:41:59,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:04,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:04,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:42:04,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:42:09,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:09,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:15,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:19,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:19,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:42:19,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:42:20,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:26,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:28,110 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:42:29,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:31,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:34,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:42:34,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:42:36,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:39,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:42,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:48,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:49,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:42:49,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:42:49,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:53,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:42:58,112 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:42:59,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:42:59,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:04,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:43:04,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:43:05,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:09,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:43:11,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:17,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:19,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:43:19,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:43:19,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:43:22,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:28,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:28,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:43:29,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:43:34,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:43:34,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:43:34,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:39,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:43:40,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:45,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:49,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:43:49,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:43:49,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:43:50,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:51,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:43:51,735 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:43:51,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:43:51,736 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:43:52,518 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:43:52,519 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:43:56,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:43:58,117 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:43:59,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:02,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:04,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:44:04,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:44:07,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:09,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:13,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:18,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:19,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:44:19,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:44:19,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:23,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:28,120 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:44:29,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:29,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:34,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:44:34,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:44:34,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:39,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:40,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:45,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:49,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:44:49,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:44:49,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:44:50,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:56,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:44:58,127 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:44:59,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:02,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:04,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:45:04,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:45:07,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:09,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:13,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:18,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:19,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:45:19,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:45:19,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:23,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:28,125 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:45:29,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:29,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:34,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:45:34,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:45:34,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:39,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:40,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:45,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:49,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:45:49,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:45:49,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:45:50,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:53,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:45:53,477 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:45:53,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:45:53,480 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:45:53,571 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:45:54,571 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:45:56,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:45:58,127 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:45:59,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:02,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:04,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:46:04,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:46:07,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:09,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:13,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:18,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:19,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:46:19,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:46:19,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:23,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:28,131 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:46:29,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:29,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:34,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:46:34,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:46:35,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:39,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:41,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:46,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:49,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:46:49,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:46:49,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:46:51,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:57,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:46:58,132 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:46:59,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:03,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:04,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:47:04,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:47:08,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:09,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:14,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:19,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:47:19,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:47:19,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:19,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:25,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:28,134 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:47:29,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:31,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:34,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:47:34,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:47:36,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:39,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:42,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:47,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:49,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:47:49,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:47:49,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:47:52,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:55,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:47:55,014 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:47:55,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:47:55,015 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:47:55,626 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:47:56,627 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:47:58,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:47:58,137 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:47:59,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:03,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:04,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:48:04,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:48:08,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:09,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:14,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:19,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:19,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:19,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:48:19,312 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:48:24,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:28,140 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:48:29,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:30,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:34,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:48:34,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:48:36,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:39,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:41,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:46,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:49,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:48:49,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:48:49,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:48:51,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:57,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:48:58,142 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:48:59,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:03,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:04,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:49:04,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:49:08,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:09,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:14,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:19,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:19,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:49:19,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:49:19,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:25,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:28,149 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:49:29,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:31,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:34,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:49:34,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:49:36,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:42,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:47,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:49,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:49:49,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:49:49,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:49:52,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:58,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:49:58,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:49:58,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:49:58,332 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:49:58,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:49:58,335 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:49:58,681 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:49:59,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:00,682 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:50:03,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:04,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:50:04,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:50:08,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:09,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:14,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:19,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:19,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:50:19,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:50:19,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:25,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:28,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:50:29,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:31,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:34,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:50:34,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:50:37,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:39,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:42,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:48,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:49,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:50:49,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:50:49,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:50:53,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:58,153 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:50:59,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:50:59,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:04,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:51:04,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:51:04,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:09,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:10,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:16,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:19,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:19,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:51:19,312 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:51:21,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:27,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:28,155 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:51:29,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:33,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:34,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:51:34,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:51:38,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:39,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:44,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:49,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:51:49,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:51:49,312 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:51:49,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:55,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:51:58,157 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:51:59,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:00,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:52:00,279 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:52:00,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:52:00,280 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:52:00,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:00,749 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:52:02,750 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:52:04,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:52:04,312 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:52:05,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:09,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:11,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:16,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:19,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:19,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:52:19,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:52:21,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:27,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:28,160 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:52:29,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:33,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:34,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:52:34,312 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:52:38,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:39,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:44,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:49,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:52:49,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:52:49,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:52:49,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:55,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:52:58,163 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:52:59,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:01,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:04,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:53:04,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:53:06,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:09,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:12,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:17,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:19,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:19,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:53:19,320 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:53:22,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:28,165 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:53:28,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:29,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:33,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:34,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:53:34,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:53:38,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:39,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:44,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:49,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:53:49,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:53:49,321 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:53:49,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:55,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:53:58,173 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:53:59,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:01,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:01,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:54:02,001 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:54:02,002 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:54:02,002 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:54:02,811 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:54:04,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:54:04,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:54:04,812 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:54:06,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:09,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:12,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:17,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:19,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:19,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:54:19,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:54:22,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:28,170 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:54:28,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:29,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:33,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:34,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:54:34,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:54:38,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:39,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:44,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:49,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:54:49,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:54:49,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:54:49,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:55,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:54:58,174 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:54:59,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:01,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:04,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:55:04,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:55:06,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:09,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:12,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:17,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:19,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:19,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:55:19,322 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:55:22,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:28,175 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:55:28,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:29,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:33,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:34,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:55:34,323 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:55:38,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:39,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:44,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:49,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:55:49,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:55:49,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:55:49,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:55,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:55:58,177 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:55:59,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:01,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:04,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:56:04,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:56:05,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:56:05,300 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:56:05,301 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:56:05,302 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:56:05,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:56:06,888 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:56:07,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:09,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:12,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:17,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:19,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:19,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:56:19,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:56:22,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:28,180 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:56:28,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:29,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:33,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:34,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:56:34,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:56:38,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:39,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:44,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:49,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:56:49,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:56:49,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:56:49,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:55,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:56:58,182 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:56:59,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:01,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:04,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:57:04,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:57:07,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:09,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:12,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:18,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:19,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:19,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:57:19,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:57:23,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:28,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:57:29,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:29,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:34,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:57:34,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:57:34,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:39,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:40,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:45,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:49,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:57:49,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:57:49,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:57:50,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:56,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:57:58,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:57:59,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:02,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:04,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:58:04,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:58:07,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 06:58:07,087 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 06:58:07,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 06:58:07,088 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 06:58:07,948 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 06:58:08,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:08,949 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 06:58:09,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:13,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:19,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:19,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:58:19,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:58:19,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:25,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:28,189 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:58:29,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:31,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:34,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:58:34,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:58:36,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:39,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:42,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:47,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:49,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:58:49,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:58:49,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:58:52,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:58,203 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:58:58,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:58:59,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:03,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:04,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:59:04,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:59:08,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:09,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:14,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:19,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:19,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:59:19,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:59:19,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:25,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:28,193 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:59:29,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:31,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:34,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:59:34,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:59:36,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:39,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:42,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:47,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:49,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 06:59:49,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 06:59:49,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 06:59:52,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:58,197 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 06:59:58,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 06:59:59,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:03,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:04,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:00:04,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:00:08,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:08,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:00:08,839 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:00:08,840 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:00:08,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:00:09,017 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:00:09,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:11,018 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:00:14,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:19,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:19,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:00:19,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:00:19,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:25,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:28,198 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:00:29,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:31,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:34,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:00:34,334 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:00:36,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:39,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:42,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:48,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:49,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:00:49,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:00:49,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:00:53,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:58,201 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:00:59,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:00:59,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:04,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:01:04,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:01:05,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:09,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:10,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:15,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:19,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:19,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:01:19,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:01:20,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:26,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:28,203 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:01:29,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:32,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:34,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:01:34,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:01:37,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:39,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:43,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:49,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:01:49,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:01:49,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:01:49,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:55,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:01:58,206 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:01:59,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:01,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:04,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:02:04,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:02:06,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:09,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:12,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:02:12,008 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:02:12,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:02:12,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:12,010 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:02:12,075 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:02:13,075 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:02:17,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:19,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:19,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:02:19,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:02:22,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:28,208 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:02:28,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:29,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:33,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:34,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:02:34,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:02:38,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:39,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:44,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:49,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:02:49,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:02:49,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:02:49,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:55,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:02:58,210 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:02:59,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:00,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:04,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:03:04,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:03:05,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:09,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:11,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:17,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:19,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:19,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:03:19,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:03:22,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:28,217 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:03:28,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:29,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:33,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:34,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:03:34,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:03:38,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:39,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:44,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:49,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:03:49,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:03:49,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:03:49,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:55,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:03:58,215 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:03:59,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:01,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:04,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:04:04,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:04:07,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:09,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:12,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:13,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:04:13,607 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:04:13,607 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:04:13,609 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:04:14,123 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:04:15,123 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:04:17,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:19,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:19,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:04:19,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:04:22,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:28,218 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:04:28,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:29,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:33,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:34,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:04:34,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:04:39,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:39,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:44,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:49,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:04:49,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:04:49,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:04:49,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:55,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:04:58,221 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:04:59,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:01,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:04,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:05:04,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:05:06,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:09,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:12,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:17,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:19,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:19,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:05:19,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:05:22,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:28,223 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:05:28,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:29,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:33,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:34,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:05:34,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:05:38,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:39,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:44,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:49,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:05:49,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:05:49,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:05:49,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:55,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:05:58,225 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:05:59,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:01,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:04,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:06:04,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:06:06,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:09,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:12,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:15,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:06:15,514 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:06:15,515 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:06:15,516 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:06:16,178 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:06:17,179 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:06:17,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:19,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:19,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:06:19,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:06:22,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:28,227 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:06:28,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:29,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:33,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:34,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:06:34,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:06:38,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:39,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:44,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:49,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:06:49,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:06:49,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:06:49,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:55,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:06:58,230 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:06:59,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:01,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:04,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:07:04,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:07:06,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:09,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:12,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:18,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:19,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:19,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:07:19,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:07:23,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:28,232 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:07:29,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:29,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:34,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:07:34,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:07:34,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:39,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:40,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:46,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:49,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:07:49,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:07:49,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:07:51,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:57,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:07:58,238 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:07:59,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:02,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:04,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:08:04,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:08:07,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:09,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:13,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:18,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:18,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:08:18,640 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:08:18,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:08:18,641 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:08:19,244 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:08:19,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:08:19,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:08:19,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:21,245 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:08:23,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:28,236 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:08:29,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:29,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:34,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:08:34,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:08:34,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:39,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:40,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:45,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:49,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:08:49,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:08:49,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:08:50,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:56,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:08:58,240 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:08:59,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:02,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:04,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:09:04,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:09:08,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:09,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:13,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:18,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:19,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:19,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:09:19,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:09:23,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:28,241 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:09:29,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:29,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:34,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:09:34,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:09:34,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:39,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:40,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:45,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:49,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:09:49,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:09:49,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:09:50,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:56,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:09:58,244 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:09:59,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:02,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:04,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:10:04,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:10:07,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:09,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:13,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:18,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:19,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:19,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:10:19,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:10:20,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:10:20,336 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:10:20,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:10:20,338 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:10:21,290 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:10:23,291 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:10:24,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:28,246 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:10:29,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:29,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:34,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:10:34,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:10:34,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:39,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:40,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:45,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:49,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:10:49,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:10:49,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:10:50,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:56,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:10:58,249 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:10:59,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:02,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:04,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:11:04,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:11:07,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:09,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:13,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:18,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:19,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:19,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:11:19,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:11:23,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:28,251 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:11:29,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:29,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:34,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:11:34,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:11:34,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:39,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:40,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:45,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:49,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:11:49,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:11:49,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:11:50,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:56,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:11:58,253 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:11:59,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:02,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:04,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:12:04,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:12:08,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:09,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:13,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:19,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:19,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:12:19,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:12:19,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:21,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:12:21,980 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:12:21,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:12:21,981 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:12:22,338 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:12:23,339 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:12:25,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:28,260 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:12:29,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:31,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:34,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:12:34,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:12:36,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:39,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:42,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:47,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:49,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:12:49,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:12:49,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:12:52,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:58,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:12:58,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:12:59,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:03,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:04,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:13:04,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:13:08,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:09,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:14,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:19,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:13:19,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:13:19,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:19,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:25,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:28,262 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:13:29,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:31,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:34,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:13:34,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:13:36,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:39,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:42,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:47,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:49,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:13:49,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:13:49,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:13:52,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:58,263 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:13:58,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:13:59,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:03,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:04,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:14:04,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:14:08,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:09,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:14,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:19,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:14:19,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:14:19,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:19,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:25,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:14:25,222 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:14:25,222 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:14:25,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:25,224 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:14:25,389 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:14:27,390 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:14:28,266 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:14:29,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:30,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:34,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:14:34,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:14:35,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:39,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:41,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:46,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:49,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:14:49,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:14:49,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:14:51,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:57,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:14:58,268 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:14:59,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:02,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:04,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:15:04,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:15:07,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:09,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:13,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:18,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:19,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:15:19,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:15:19,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:23,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:28,270 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:15:29,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:29,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:34,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:15:34,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:15:34,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:39,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:40,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:45,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:49,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:15:49,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:15:49,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:15:50,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:56,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:15:58,272 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:15:59,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:02,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:04,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:16:04,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:16:07,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:09,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:13,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:18,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:19,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:16:19,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:16:19,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:23,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:26,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:16:26,950 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:16:26,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:16:26,952 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:16:27,436 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:16:28,275 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:16:29,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:29,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:29,437 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:16:34,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:16:34,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:16:34,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:39,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:40,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:45,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:49,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:16:49,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:16:49,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:16:50,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:56,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:16:58,278 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:16:59,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:02,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:04,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:17:04,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:17:08,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:09,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:13,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:19,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:17:19,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:17:19,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:19,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:25,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:28,284 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:17:29,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:31,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:34,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:17:34,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:17:36,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:39,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:42,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:47,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:49,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:17:49,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:17:49,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:17:52,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:58,282 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:17:58,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:17:59,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:03,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:04,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:18:04,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:18:08,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:09,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:14,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:19,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:18:19,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:18:19,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:19,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:25,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:28,286 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:18:28,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:18:28,655 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:18:28,655 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:18:28,656 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:18:29,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:29,485 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:18:30,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:31,487 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:18:34,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:18:34,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:18:35,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:39,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:41,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:46,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:49,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:18:49,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:18:49,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:18:51,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:57,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:18:58,286 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:18:59,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:03,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:04,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:19:04,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:19:09,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:09,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:14,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:19,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:19:19,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:19:19,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:19,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:25,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:28,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:19:29,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:30,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:34,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:19:34,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:19:35,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:39,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:41,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:46,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:49,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:19:49,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:19:49,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:19:51,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:57,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:19:58,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:19:59,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:02,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:04,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:20:04,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:20:07,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:09,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:13,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:19,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:20:19,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:20:19,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:19,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:25,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:28,295 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:20:29,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:30,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:20:30,261 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:20:30,261 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:20:30,262 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:20:30,544 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:20:31,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:31,544 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:20:34,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:20:34,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:20:36,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:39,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:42,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:48,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:49,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:20:49,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:20:49,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:20:53,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:58,298 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:20:59,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:20:59,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:04,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:21:04,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:21:05,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:09,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:10,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:16,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:19,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:21:19,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:21:19,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:21,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:27,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:28,302 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:21:29,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:33,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:34,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:21:34,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:21:38,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:39,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:44,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:49,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:21:49,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:21:49,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:21:49,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:55,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:21:58,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:21:59,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:00,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:04,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:22:04,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:22:05,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:09,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:11,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:16,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:19,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:22:19,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:22:19,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:21,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:27,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:28,307 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:22:29,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:33,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:22:33,457 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:22:33,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:22:33,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:33,460 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:22:33,599 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:22:34,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:22:34,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:22:35,600 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:22:38,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:39,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:44,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:49,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:22:49,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:22:49,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:22:49,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:54,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:22:58,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:22:59,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:00,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:04,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:23:04,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:23:05,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:09,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:11,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:16,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:19,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:23:19,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:23:19,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:21,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:27,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:28,311 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:23:29,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:33,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:34,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:23:34,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:23:38,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:39,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:44,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:49,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:23:49,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:23:49,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:23:49,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:54,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:23:58,313 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:23:59,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:00,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:04,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:24:04,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:24:05,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:09,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:11,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:16,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:19,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:24:19,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:24:19,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:21,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:27,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:28,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:24:29,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:33,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:34,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:24:34,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:24:35,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:24:35,218 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:24:35,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:24:35,220 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:24:35,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:24:37,655 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:24:39,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:39,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:44,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:49,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:24:49,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:24:49,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:24:49,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:55,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:24:58,318 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:24:59,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:01,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:04,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:25:04,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:25:06,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:09,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:12,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:17,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:19,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:25:19,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:25:19,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:22,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:28,321 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:25:28,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:29,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:33,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:34,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:25:34,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:25:38,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:39,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:44,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:49,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:25:49,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:25:49,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:25:49,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:55,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:25:58,323 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:25:59,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:00,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:04,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:26:04,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:26:05,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:09,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:11,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:16,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:19,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:26:19,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:26:19,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:21,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:27,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:28,329 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:26:29,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:32,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:34,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:26:34,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:26:36,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:26:36,487 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:26:36,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:26:36,488 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:26:36,700 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:26:37,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:26:38,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:39,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:43,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:48,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:49,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:26:49,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:26:49,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:26:54,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:26:58,328 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:26:59,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:00,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:04,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:27:04,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:27:05,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:09,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:11,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:17,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:19,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:27:19,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:27:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:22,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:28,332 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:27:28,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:29,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:33,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:34,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:27:34,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:27:39,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:39,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:45,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:49,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:27:49,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:27:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:27:50,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:56,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:27:58,333 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:27:59,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:01,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:04,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:28:04,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:28:07,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:09,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:12,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:17,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:19,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:28:19,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:28:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:22,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:28,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:28:28,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:29,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:33,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:34,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:28:34,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:28:38,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:39,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:39,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:28:39,504 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:28:39,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:28:39,505 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:28:39,752 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:28:41,753 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:28:44,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:49,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:28:49,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:28:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:28:49,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:55,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:28:58,339 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:28:59,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:01,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:04,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:29:04,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:29:07,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:09,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:13,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:18,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:19,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:29:19,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:29:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:23,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:28,341 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:29:29,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:29,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:34,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:29:34,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:29:34,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:39,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:40,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:45,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:49,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:29:49,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:29:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:29:51,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:56,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:29:58,343 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:29:59,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:02,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:04,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:30:04,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:30:07,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:09,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:13,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:18,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:19,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:30:19,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:30:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:24,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:28,346 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:30:29,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:30,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:34,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:30:34,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:30:35,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:39,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:40,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:30:40,976 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:30:40,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:30:40,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:40,977 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:30:41,802 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:30:43,804 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:30:46,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:49,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:30:49,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:30:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:30:51,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:57,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:30:58,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:30:59,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:03,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:04,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:31:04,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:31:09,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:09,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:14,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:19,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:31:19,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:31:19,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:19,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:25,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:28,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:31:29,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:31,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:34,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:31:34,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:31:37,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:39,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:43,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:48,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:49,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:31:49,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:31:49,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:54,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:31:58,355 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:31:59,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:31:59,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:04,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:32:04,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:32:04,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:09,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:32:10,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:15,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:19,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:32:19,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:32:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:32:20,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:26,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:28,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:32:29,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:32:32,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:34,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:32:34,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:32:37,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:39,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:32:42,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:32:42,451 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:32:42,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:32:42,452 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:32:42,873 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:32:43,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:43,874 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:32:48,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:49,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:32:49,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:32:49,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:32:53,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:58,358 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:32:59,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:32:59,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:04,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:33:04,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:33:04,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:09,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:10,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:15,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:19,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:33:19,388 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:33:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:20,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:26,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:28,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:33:29,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:32,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:34,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:33:34,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:33:38,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:39,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:43,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:48,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:49,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:33:49,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:33:49,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:54,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:33:58,363 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:33:59,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:33:59,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:04,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:34:04,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:34:05,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:09,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:34:10,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:15,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:19,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:34:19,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:34:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:34:20,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:26,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:28,366 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:34:29,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:34:31,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:34,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:34:34,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:34:36,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:39,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:34:42,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:45,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:34:45,623 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:34:45,624 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:34:45,624 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:34:45,939 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:34:47,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:47,940 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:34:49,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:34:49,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:34:49,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:34:53,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:58,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:34:59,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:34:59,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:04,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:35:04,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:35:04,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:09,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:09,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:14,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:19,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:35:19,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:35:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:19,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:25,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:28,371 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:35:29,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:30,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:34,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:35:34,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:35:35,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:39,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:41,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:46,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:49,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:35:49,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:35:49,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:35:51,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:57,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:35:58,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:35:59,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:02,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:04,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:36:04,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:36:08,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:09,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:13,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:19,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:36:19,391 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:36:19,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:19,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:25,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:28,375 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:36:29,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:30,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:34,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:36:34,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:36:35,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:39,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:41,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:46,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:47,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:36:47,181 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:36:47,181 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:36:47,183 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:36:47,990 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:36:49,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:36:49,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:36:49,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:36:49,991 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:36:51,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:57,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:36:58,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:36:59,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:03,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:04,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:37:04,392 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:37:09,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:09,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:15,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:19,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:37:19,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:37:19,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:21,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:26,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:28,382 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:37:29,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:31,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:34,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:37:34,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:37:37,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:39,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:42,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:47,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:49,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:37:49,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:37:49,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:37:53,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:58,384 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:37:59,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:37:59,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:04,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:38:04,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:38:04,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:09,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:09,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:14,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:19,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:38:19,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:38:19,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:20,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:26,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:28,386 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:38:29,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:31,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:34,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:38:34,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:38:37,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:39,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:42,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:48,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:48,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:38:48,669 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:38:48,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:38:48,670 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:38:49,044 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:38:49,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:38:49,393 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:38:49,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:38:50,045 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:38:53,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:58,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:38:59,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:38:59,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:04,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:39:04,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:39:04,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:09,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:09,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:14,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:19,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:39:19,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:39:19,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:19,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:25,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:28,391 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:39:29,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:30,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:34,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:39:34,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:39:36,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:39,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:41,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:46,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:49,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:39:49,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:39:49,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:39:52,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:57,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:39:58,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:39:59,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:03,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:04,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:40:04,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:40:09,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:10,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:14,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:19,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:40:19,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:40:20,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:20,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:25,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:28,400 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:40:30,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:31,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:34,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:40:34,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:40:36,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:40,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:42,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:47,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:49,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:40:49,395 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:40:50,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:40:51,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:40:51,794 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:40:51,794 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:40:51,795 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:40:52,100 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:40:52,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:40:54,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:40:58,398 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:40:58,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:00,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:03,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:04,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:41:04,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:41:08,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:10,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:14,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:19,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:41:19,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:41:20,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:20,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:25,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:28,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:41:30,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:31,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:34,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:41:34,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:41:37,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:40,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:43,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:48,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:49,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:41:49,414 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:41:50,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:41:53,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:41:58,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:41:59,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:00,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:04,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:42:04,414 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:42:04,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:09,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:10,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:15,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:19,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:42:19,414 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:42:20,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:21,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:26,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:28,406 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:42:30,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:32,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:34,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:42:34,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:42:37,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:40,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:42,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:47,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:49,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:42:49,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:42:50,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:42:53,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:42:53,201 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:42:53,201 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:42:53,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:42:53,202 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:42:54,147 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:42:56,148 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:42:58,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:42:58,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:00,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:03,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:04,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:43:04,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:43:09,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:10,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:14,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:19,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:43:19,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:43:20,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:20,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:25,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:28,411 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:43:30,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:31,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:34,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:43:34,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:43:36,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:40,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:42,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:47,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:49,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:43:49,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:43:50,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:43:53,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:43:58,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:43:59,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:00,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:04,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:44:04,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:44:04,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:09,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:10,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:15,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:19,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:44:19,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:44:20,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:21,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:26,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:28,415 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:44:30,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:32,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:34,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:44:34,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:44:37,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:40,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:43,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:48,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:49,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:44:49,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:44:50,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:44:54,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:44:54,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:44:54,660 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:44:54,660 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:44:54,661 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:44:55,195 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:44:56,196 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:44:58,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:45:00,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:00,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:04,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:45:04,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:45:05,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:10,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:10,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:15,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:19,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:45:19,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:45:20,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:21,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:26,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:28,421 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:45:30,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:32,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:34,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:45:34,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:45:37,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:40,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:43,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:48,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:49,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:45:49,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:45:50,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:45:54,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:45:58,425 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:46:00,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:00,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:04,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:46:04,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:46:05,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:10,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:10,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:16,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:19,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:46:19,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:46:20,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:22,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:27,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:28,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:46:30,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:33,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:34,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:46:34,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:46:39,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:40,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:45,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:49,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:46:49,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:46:50,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:50,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:46:56,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:46:57,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:46:57,811 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:46:57,811 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:46:57,812 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:46:58,253 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:46:58,428 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:46:59,254 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:47:00,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:01,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:04,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:47:04,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:47:06,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:10,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:11,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:16,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:19,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:47:19,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:47:20,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:22,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:27,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:28,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:47:30,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:32,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:34,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:47:34,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:47:38,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:40,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:44,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:49,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:47:49,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:47:49,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:50,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:47:55,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:47:58,433 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:48:00,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:00,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:04,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:48:04,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:48:05,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:10,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:11,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:16,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:19,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:48:19,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:48:20,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:22,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:27,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:28,435 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:48:30,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:33,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:34,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:48:34,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:48:39,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:40,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:44,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:49,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:48:49,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:48:49,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:50,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:48:55,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:48:58,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:48:59,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:48:59,574 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:48:59,575 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:48:59,575 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:49:00,307 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:49:00,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:00,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:01,307 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:49:04,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:49:04,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:49:06,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:10,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:11,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:16,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:19,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:49:19,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:49:20,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:22,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:28,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:28,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:49:30,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:34,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:34,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:49:34,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:49:39,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:40,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:45,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:49,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:49:49,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:49:50,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:50,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:49:56,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:49:58,513 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:50:00,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:01,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:04,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:50:04,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:50:07,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:10,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:12,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:18,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:19,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:50:19,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:50:20,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:24,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:28,515 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:50:29,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:30,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:34,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:50:34,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:50:34,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:40,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:40,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:45,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:49,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:50:49,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:50:50,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:50,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:50:56,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:50:58,517 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:51:00,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:01,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:01,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:51:01,180 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:51:01,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:51:01,181 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:51:01,355 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:51:03,356 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:51:04,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:51:04,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:51:06,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:10,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:12,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:17,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:19,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:51:19,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:51:20,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:23,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:28,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:28,520 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:51:30,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:34,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:34,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:51:34,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:51:39,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:40,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:45,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:49,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:51:49,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:51:50,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:50,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:51:56,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:51:58,522 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:52:00,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:01,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:04,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:52:04,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:52:07,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:10,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:12,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:18,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:19,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:52:19,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:52:20,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:24,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:28,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:52:29,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:30,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:34,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:52:34,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:52:34,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:40,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:40,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:46,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:49,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:52:49,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:52:50,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:52:51,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:57,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:52:58,527 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:53:00,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:02,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:04,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:53:04,310 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:53:04,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:53:04,311 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:53:04,400 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:53:04,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:53:04,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:53:05,401 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:53:08,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:10,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:14,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:19,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:19,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:53:19,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:53:20,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:25,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:28,531 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:53:30,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:30,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:34,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:53:34,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:53:35,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:40,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:41,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:46,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:49,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:53:49,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:53:50,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:53:52,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:58,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:53:58,532 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:54:00,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:03,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:04,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:54:04,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:54:08,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:10,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:14,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:19,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:19,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:54:19,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:54:20,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:25,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:28,540 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:54:30,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:30,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:34,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:54:34,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:54:36,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:40,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:41,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:46,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:54:49,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:54:50,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:54:52,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:57,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:54:58,537 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:55:00,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:03,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:04,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:55:04,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:55:06,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:55:06,184 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:55:06,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:55:06,185 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:55:06,462 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:55:07,463 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:55:08,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:10,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:14,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:19,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:19,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:55:19,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:55:20,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:25,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:28,541 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:55:30,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:30,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:34,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:55:34,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:55:35,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:40,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:41,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:46,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:49,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:55:49,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:55:50,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:55:52,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:57,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:55:58,543 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:56:00,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:02,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:04,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:56:04,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:56:08,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:10,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:14,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:19,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:19,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:56:19,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:56:20,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:25,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:28,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:56:30,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:30,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:34,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:56:34,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:56:35,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:40,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:41,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:46,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:49,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:56:49,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:56:50,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:56:52,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:58,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:56:58,548 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:57:00,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:03,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:04,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:57:04,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:57:07,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:57:07,687 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:57:07,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:57:07,688 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:57:08,508 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:57:08,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:09,508 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:57:10,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:14,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:19,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:19,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:57:19,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:57:20,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:24,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:28,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:57:29,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:30,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:34,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:57:34,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:57:34,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:40,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:40,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:46,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:49,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:57:49,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:57:50,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:57:52,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:58,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:57:58,554 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:58:00,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:03,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:04,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:58:04,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:58:09,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:10,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:15,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:19,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:58:19,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:58:20,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:20,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:26,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:28,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:58:30,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:32,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:34,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:58:34,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:58:38,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:40,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:43,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:49,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:49,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:58:49,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:58:50,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:58:55,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:58:58,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:59:00,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:00,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:04,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:59:04,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:59:06,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:09,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 07:59:09,335 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 07:59:09,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 07:59:09,336 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 07:59:09,551 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 07:59:10,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:11,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:11,552 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 07:59:17,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:59:19,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:59:20,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:23,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:28,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:28,560 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 07:59:30,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:33,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:34,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:59:34,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:59:39,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:40,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:45,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:49,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 07:59:49,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 07:59:50,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:50,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 07:59:56,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 07:59:58,564 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:00:00,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:01,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:04,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:00:04,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:00:06,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:10,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:12,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:18,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:19,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:00:19,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:00:20,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:23,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:28,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:28,567 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:00:30,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:34,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:00:34,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:00:34,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:40,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:40,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:46,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:49,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:00:49,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:00:50,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:00:51,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:57,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:00:58,569 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:01:00,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:02,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:04,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:01:04,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:01:08,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:10,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:12,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:01:12,731 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:01:12,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:01:12,732 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:01:13,617 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:01:13,618 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:01:13,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:19,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:01:19,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:01:19,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:20,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:25,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:28,571 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:01:30,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:30,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:34,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:01:34,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:01:35,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:40,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:41,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:47,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:49,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:01:49,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:01:50,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:01:52,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:58,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:01:58,574 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:02:00,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:03,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:04,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:02:04,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:02:08,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:10,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:13,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:18,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:19,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:02:19,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:02:20,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:24,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:28,576 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:02:29,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:30,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:34,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:02:34,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:02:34,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:40,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:40,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:46,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:49,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:02:49,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:02:50,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:02:51,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:57,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:02:58,580 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:03:00,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:02,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:04,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:03:04,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:03:08,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:10,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:14,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:14,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:03:14,624 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:03:14,624 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:03:14,625 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:03:14,677 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:03:15,678 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:03:19,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:03:19,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:03:19,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:20,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:25,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:28,586 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:03:30,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:30,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:34,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:03:34,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:03:35,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:40,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:41,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:47,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:49,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:03:49,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:03:50,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:03:52,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:57,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:03:58,585 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:04:00,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:02,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:04,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:04:04,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:04:08,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:10,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:14,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:19,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:04:19,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:04:19,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:20,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:25,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:28,592 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:04:30,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:30,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:34,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:04:34,436 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:04:36,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:40,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:41,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:47,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:49,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:04:49,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:04:50,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:04:52,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:58,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:04:58,591 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:05:00,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:04,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:05:04,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:05:04,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:10,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:10,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:16,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:05:16,326 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:05:16,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:05:16,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:16,328 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:05:16,738 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:05:17,739 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:05:19,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:05:19,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:05:20,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:21,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:27,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:28,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:05:30,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:32,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:34,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:05:34,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:05:38,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:40,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:43,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:49,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:05:49,437 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:05:49,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:50,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:05:54,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:05:58,596 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:06:00,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:00,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:04,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:06:04,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:06:06,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:10,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:11,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:16,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:19,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:06:19,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:06:20,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:22,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:27,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:28,601 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:06:30,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:32,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:34,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:06:34,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:06:38,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:40,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:43,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:48,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:49,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:06:49,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:06:50,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:06:53,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:06:58,603 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:06:59,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:00,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:04,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:07:04,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:07:05,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:10,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:10,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:16,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:19,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:07:19,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:07:19,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:07:19,744 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:07:19,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:07:19,745 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:07:19,804 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:07:20,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:21,804 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:07:22,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:27,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:28,606 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:07:30,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:32,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:34,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:07:34,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:07:38,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:40,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:43,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:49,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:07:49,439 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:07:49,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:50,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:07:54,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:07:58,643 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:08:00,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:00,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:04,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:08:04,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:08:06,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:10,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:11,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:16,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:19,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:08:19,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:08:20,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:22,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:27,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:28,612 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:08:30,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:32,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:34,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:08:34,442 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:08:38,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:40,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:43,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:49,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:08:49,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:08:49,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:50,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:08:54,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:08:58,617 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:08:59,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:00,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:04,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:09:04,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:09:04,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:10,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:10,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:16,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:19,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:09:19,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:09:20,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:22,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:09:22,519 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:09:22,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:09:22,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:22,522 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:09:22,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:09:23,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:09:27,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:28,617 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:09:30,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:32,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:34,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:09:34,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:09:38,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:40,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:43,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:48,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:49,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:09:49,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:09:50,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:09:53,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:09:58,622 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:09:59,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:00,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:04,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:10:04,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:10:05,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:10,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:10,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:15,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:19,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:10:19,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:10:20,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:21,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:26,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:28,623 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:10:30,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:32,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:34,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:10:34,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:10:38,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:40,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:43,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:49,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:10:49,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:10:49,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:50,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:10:54,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:10:58,627 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:11:00,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:00,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:04,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:11:04,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:11:06,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:10,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:11,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:16,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:19,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:11:19,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:11:20,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:22,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:24,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:11:24,562 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:11:24,562 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:11:24,563 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:11:24,940 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:11:25,941 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:11:27,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:28,628 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:11:30,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:32,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:34,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:11:34,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:11:38,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:40,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:43,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:49,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:11:49,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:11:49,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:50,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:11:54,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:11:58,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:11:59,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:00,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:04,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:12:04,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:12:05,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:10,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:10,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:15,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:19,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:12:19,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:12:20,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:21,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:26,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:28,632 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:12:30,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:32,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:34,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:12:34,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:12:38,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:40,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:43,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:48,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:49,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:12:49,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:12:50,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:12:53,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:12:58,654 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:12:58,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:00,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:04,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:13:04,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:13:04,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:09,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:10,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:15,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:19,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:13:19,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:13:20,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:21,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:26,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:28,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:13:28,124 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:13:28,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:13:28,125 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:13:28,638 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:13:29,003 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:13:30,004 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:13:30,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:31,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:34,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:13:34,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:13:37,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:40,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:42,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:47,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:49,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:13:49,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:13:50,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:13:53,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:13:58,641 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:13:58,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:00,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:03,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:04,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:14:04,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:14:09,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:10,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:14,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:19,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:14:19,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:14:20,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:20,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:25,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:28,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:14:30,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:30,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:34,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:14:34,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:14:36,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:40,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:41,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:47,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:49,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:14:49,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:14:50,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:14:53,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:14:58,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:14:58,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:00,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:03,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:04,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:15:04,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:15:08,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:10,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:14,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:19,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:15:19,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:15:20,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:20,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:25,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:28,648 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:15:30,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:15:30,379 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:15:30,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:15:30,381 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:15:30,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:31,089 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:15:31,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:32,090 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:15:34,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:15:34,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:15:36,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:40,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:41,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:46,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:49,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:15:49,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:15:50,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:15:52,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:57,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:15:58,651 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:16:00,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:03,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:04,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:16:04,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:16:08,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:10,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:14,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:19,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:16:19,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:16:20,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:20,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:25,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:28,653 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:16:30,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:31,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:34,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:16:34,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:16:37,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:40,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:42,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:47,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:49,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:16:49,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:16:50,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:16:53,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:16:58,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:16:58,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:00,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:03,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:04,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:17:04,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:17:09,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:10,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:15,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:19,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:17:19,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:17:20,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:21,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:26,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:28,671 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:17:30,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:32,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:17:32,028 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:17:32,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:17:32,029 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:17:32,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:32,157 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:17:34,158 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:17:34,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:17:34,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:17:37,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:40,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:42,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:48,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:49,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:17:49,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:17:50,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:17:54,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:17:58,660 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:18:00,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:00,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:04,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:18:04,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:18:05,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:10,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:11,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:16,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:19,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:18:19,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:18:20,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:21,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:27,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:28,663 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:18:30,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:32,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:34,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:18:34,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:18:37,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:40,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:43,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:48,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:49,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:18:49,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:18:50,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:18:53,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:18:58,664 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:18:58,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:00,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:04,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:04,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:19:04,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:19:09,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:10,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:15,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:19,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:19:19,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:19:20,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:20,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:26,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:28,667 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:19:30,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:32,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:34,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:19:34,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:19:35,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:19:35,505 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:19:35,506 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:19:35,507 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:19:36,216 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:19:36,216 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:19:37,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:40,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:43,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:48,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:49,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:19:49,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:19:50,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:19:53,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:19:58,669 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:19:58,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:00,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:04,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:04,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:20:04,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:20:09,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:10,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:15,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:19,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:20:19,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:20:20,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:20,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:26,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:28,671 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:20:30,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:31,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:34,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:20:34,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:20:36,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:40,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:42,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:47,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:49,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:20:49,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:20:50,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:20:52,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:58,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:20:58,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:21:00,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:04,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:04,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:21:04,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:21:09,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:10,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:14,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:19,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:21:19,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:21:19,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:20,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:25,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:28,677 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:21:30,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:30,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:34,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:21:34,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:21:35,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:37,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:21:37,093 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:21:37,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:21:37,094 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:21:37,263 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:21:38,264 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:21:40,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:41,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:46,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:49,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:21:49,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:21:50,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:21:51,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:57,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:21:58,696 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:22:00,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:03,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:04,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:22:04,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:22:09,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:10,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:14,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:19,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:19,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:22:19,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:22:20,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:24,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:28,681 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:22:30,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:30,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:34,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:22:34,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:22:35,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:40,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:41,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:46,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:49,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:22:49,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:22:50,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:22:51,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:57,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:22:58,685 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:23:00,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:02,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:04,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:23:04,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:23:07,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:10,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:13,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:18,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:19,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:23:19,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:23:20,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:23,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:28,687 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:23:28,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:30,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:34,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:34,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:23:34,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:23:39,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:39,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:23:39,117 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:23:39,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:23:39,123 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:23:39,323 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:23:40,324 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:23:40,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:45,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:49,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:23:49,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:23:50,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:50,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:23:56,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:23:58,690 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:24:00,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:02,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:04,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:24:04,475 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:24:07,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:10,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:13,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:18,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:19,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:24:19,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:24:20,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:23,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:28,692 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:24:28,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:30,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:34,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:34,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:24:34,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:24:39,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:40,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:45,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:49,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:24:49,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:24:50,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:50,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:24:56,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:24:58,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:25:00,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:01,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:04,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:25:04,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:25:06,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:10,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:12,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:18,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:19,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:25:19,476 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:25:20,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:23,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:28,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:25:28,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:30,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:34,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:34,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:25:34,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:25:40,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:40,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:42,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:25:42,311 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:25:42,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:25:42,312 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:25:42,388 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:25:44,389 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:25:45,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:49,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:25:49,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:25:50,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:25:51,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:56,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:25:58,699 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:26:00,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:01,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:04,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:26:04,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:26:07,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:10,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:13,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:18,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:19,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:26:19,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:26:20,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:24,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:28,707 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:26:29,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:30,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:34,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:26:34,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:26:35,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:40,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:40,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:45,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:49,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:26:49,478 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:26:50,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:26:51,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:56,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:26:58,705 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:27:00,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:01,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:04,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:27:04,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:27:07,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:10,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:12,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:18,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:19,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:27:19,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:27:20,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:23,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:28,708 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:27:28,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:30,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:34,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:34,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:27:34,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:27:40,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:40,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:43,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:27:43,897 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:27:43,897 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:27:43,898 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:27:44,453 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:27:46,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:46,454 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:27:49,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:27:49,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:27:50,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:27:52,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:58,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:27:58,710 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:28:00,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:03,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:04,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:28:04,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:28:09,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:10,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:14,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:19,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:28:19,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:28:19,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:20,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:25,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:28,713 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:28:30,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:31,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:34,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:28:34,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:28:36,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:40,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:41,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:47,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:49,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:28:49,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:28:50,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:28:52,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:58,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:28:58,715 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:29:00,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:03,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:04,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:29:04,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:29:09,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:10,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:14,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:19,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:29:19,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:29:19,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:20,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:25,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:28,717 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:29:30,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:31,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:34,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:29:34,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:29:37,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:40,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:43,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:45,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:29:45,692 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:29:45,692 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:29:45,693 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:29:46,524 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:29:46,524 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:29:48,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:49,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:29:49,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:29:50,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:29:54,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:29:58,719 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:29:59,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:00,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:04,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:30:04,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:30:05,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:10,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:10,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:15,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:19,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:30:19,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:30:20,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:21,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:26,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:28,723 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:30:30,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:32,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:34,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:30:34,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:30:38,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:40,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:44,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:49,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:49,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:30:49,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:30:50,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:30:54,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:30:58,730 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:30:59,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:00,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:04,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:31:04,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:31:05,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:10,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:10,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:15,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:19,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:31:19,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:31:20,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:21,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:26,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:28,727 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:31:30,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:31,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:34,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:31:34,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:31:37,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:40,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:43,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:47,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:31:47,342 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:31:47,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:31:47,344 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:31:47,574 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:31:48,575 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:31:49,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:49,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:31:49,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:31:50,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:31:54,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:31:58,730 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:31:59,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:00,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:04,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:32:04,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:32:05,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:10,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:10,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:16,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:19,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:32:19,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:32:20,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:22,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:27,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:28,733 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:32:30,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:32,920 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:34,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:32:34,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:32:38,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:40,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:43,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:49,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:32:49,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:32:49,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:50,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:32:54,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:32:58,736 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:33:00,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:00,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:04,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:33:04,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:33:06,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:10,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:11,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:16,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:19,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:33:19,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:33:20,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:21,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:27,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:28,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:33:30,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:33,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:34,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:33:34,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:33:38,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:40,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:44,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:49,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:33:49,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:33:50,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:33:50,556 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:33:50,556 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:33:50,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:50,558 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:33:50,626 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:33:50,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:33:52,627 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:33:55,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:33:58,743 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:34:00,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:01,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:04,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:34:04,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:34:06,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:10,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:11,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:16,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:19,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:34:19,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:34:20,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:22,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:27,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:28,742 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:34:30,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:33,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:34,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:34:34,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:34:38,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:40,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:44,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:49,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:34:49,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:34:50,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:50,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:34:55,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:34:58,745 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:35:00,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:01,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:04,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:35:04,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:35:06,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:10,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:11,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:16,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:19,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:35:19,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:35:20,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:22,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:28,747 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:35:28,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:30,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:33,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:34,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:35:34,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:35:38,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:40,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:44,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:49,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:35:49,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:35:50,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:50,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:35:52,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:35:52,334 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:35:52,334 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:35:52,335 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:35:52,676 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:35:54,677 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:35:56,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:35:58,752 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:36:00,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:01,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:04,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:36:04,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:36:06,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:10,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:11,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:17,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:19,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:36:19,488 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:36:20,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:23,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:28,751 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:36:29,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:30,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:34,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:36:34,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:36:35,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:40,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:40,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:46,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:49,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:36:49,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:36:50,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:36:52,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:36:58,755 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:36:58,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:00,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:03,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:04,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:37:04,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:37:09,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:10,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:14,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:19,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:37:19,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:37:20,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:20,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:25,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:28,756 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:37:30,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:31,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:34,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:37:34,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:37:36,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:40,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:41,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:46,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:49,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:37:49,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:37:50,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:37:52,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:37:53,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:37:53,817 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:37:53,817 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:37:53,818 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:37:54,724 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:37:54,724 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:37:58,757 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:37:58,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:00,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:03,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:04,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:38:04,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:38:08,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:10,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:14,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:19,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:38:19,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:38:20,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:20,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:25,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:28,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:38:30,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:31,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:34,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:38:34,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:38:36,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:40,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:42,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:48,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:49,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:38:49,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:38:50,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:38:54,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:38:58,762 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:39:00,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:00,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:04,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:39:04,490 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:39:06,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:10,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:11,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:16,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:19,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:39:19,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:39:20,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:22,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:27,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:28,764 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:39:30,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:32,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:34,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:39:34,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:39:37,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:40,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:43,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:48,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:49,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:39:49,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:39:50,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:39:54,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:39:57,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:39:57,088 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:39:57,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:39:57,089 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:39:57,773 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:39:58,767 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:39:58,774 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:40:00,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:00,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:04,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:40:04,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:40:06,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:10,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:11,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:17,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:19,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:40:19,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:40:20,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:23,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:28,815 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:40:29,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:30,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:34,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:40:34,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:40:35,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:40,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:40,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:46,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:49,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:40:49,492 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:40:50,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:40:51,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:57,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:40:58,771 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:41:00,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:02,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:04,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:41:04,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:41:08,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:10,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:13,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:19,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:41:19,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:41:19,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:20,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:24,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:28,775 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:41:30,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:30,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:34,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:41:34,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:41:36,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:40,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:41,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:46,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:49,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:41:49,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:41:50,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:41:52,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:57,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:41:58,776 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:41:58,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:41:58,910 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:41:58,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:41:58,912 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:41:59,835 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:42:00,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:00,836 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:42:03,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:04,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:42:04,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:42:10,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:10,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:15,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:19,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:42:19,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:42:20,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:20,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:26,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:28,779 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:42:30,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:31,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:34,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:42:34,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:42:37,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:40,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:43,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:49,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:49,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:42:49,493 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:42:50,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:42:55,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:42:58,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:43:00,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:00,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:04,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:43:04,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:43:06,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:10,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:12,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:18,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:19,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:43:19,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:43:20,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:24,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:28,783 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:43:29,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:30,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:34,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:43:34,494 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:43:35,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:40,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:41,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:47,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:49,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:43:49,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:43:50,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:43:53,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:58,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:43:58,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:44:00,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:44:00,625 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:44:00,625 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:44:00,626 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:44:00,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:00,883 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:44:02,884 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:44:03,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:04,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:44:04,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:44:09,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:10,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:14,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:19,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:44:19,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:44:19,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:20,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:25,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:28,788 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:44:30,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:30,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:34,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:44:34,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:44:36,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:40,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:42,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:47,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:49,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:44:49,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:44:50,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:44:52,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:57,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:44:58,794 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:45:00,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:03,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:04,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:45:04,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:45:09,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:10,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:15,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:19,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:45:19,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:45:20,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:21,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:26,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:28,792 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:45:30,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:32,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:34,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:45:34,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:45:37,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:40,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:43,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:48,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:49,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:45:49,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:45:50,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:45:54,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:45:58,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:46:00,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:00,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:03,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:46:03,786 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:46:03,786 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:46:03,787 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:46:03,944 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:46:04,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:46:04,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:46:04,944 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:46:06,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:10,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:11,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:17,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:19,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:46:19,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:46:20,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:23,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:28,797 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:46:29,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:30,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:34,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:46:34,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:46:35,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:40,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:40,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:46,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:49,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:46:49,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:46:50,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:46:52,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:57,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:46:58,799 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:47:00,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:02,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:04,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:47:04,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:47:08,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:10,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:14,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:19,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:47:19,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:47:20,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:20,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:25,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:28,802 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:47:30,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:31,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:34,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:47:34,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:47:37,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:40,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:42,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:47,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:49,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:47:49,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:47:50,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:47:53,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:47:58,804 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:47:59,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:00,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:04,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:48:04,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:48:05,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:48:05,443 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:48:05,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:48:05,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:05,445 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:48:05,999 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:48:07,000 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:48:10,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:16,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:19,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:48:19,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:48:20,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:22,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:27,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:28,807 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:48:30,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:33,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:34,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:48:34,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:48:39,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:40,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:45,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:49,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:48:49,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:48:50,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:48:51,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:56,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:48:58,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:49:00,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:02,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:04,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:49:04,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:49:07,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:10,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:12,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:17,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:19,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:49:19,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:49:20,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:23,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:28,815 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:49:29,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:30,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:34,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:49:34,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:49:35,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:40,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:40,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:45,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:49,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:49:49,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:49:50,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:49:51,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:56,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:49:58,814 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:50:00,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:02,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:04,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:50:04,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:50:06,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:50:06,938 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:50:06,938 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:50:06,940 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:50:07,057 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:50:07,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:09,058 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:50:10,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:12,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:18,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:19,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:50:19,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:50:20,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:23,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:28,816 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:50:29,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:30,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:34,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:50:34,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:50:35,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:40,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:40,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:45,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:49,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:50:49,516 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:50:50,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:50:51,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:56,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:50:58,820 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:51:00,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:02,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:04,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:51:04,519 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:51:07,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:10,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:12,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:17,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:21,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:51:21,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:51:21,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:22,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:28,821 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:51:28,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:31,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:33,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:36,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:51:36,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:51:39,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:41,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:44,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:50,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:51,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:51:51,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:51:51,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:51:56,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:51:58,823 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:52:01,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:02,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:06,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:52:06,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:52:07,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:10,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:52:10,263 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:52:10,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:52:10,265 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:52:11,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:11,114 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:52:12,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:13,115 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:52:17,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:21,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:52:21,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:52:21,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:23,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:28,825 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:52:29,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:31,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:34,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:36,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:52:36,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:52:40,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:41,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:46,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:51,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:52:51,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:52:51,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:52:52,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:57,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:52:58,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:53:01,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:02,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:06,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:53:06,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:53:08,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:11,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:14,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:20,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:21,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:53:21,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:53:21,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:25,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:28,830 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:53:31,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:31,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:36,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:53:36,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:53:37,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:41,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:42,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:47,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:51,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:53:51,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:53:51,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:53:53,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:53:58,833 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:53:59,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:01,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:04,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:06,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:54:06,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:54:09,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:11,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:11,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:54:11,951 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:54:11,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:54:11,952 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:54:12,165 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:54:13,165 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:54:15,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:21,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:54:21,017 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:54:21,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:21,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:26,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:28,835 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:54:31,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:32,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:36,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:54:36,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:54:37,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:41,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:43,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:49,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:51,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:54:51,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:54:51,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:54:55,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:54:58,838 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:55:01,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:01,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:06,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:55:06,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:55:07,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:11,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:12,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:17,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:21,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:55:21,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:55:21,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:22,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:28,840 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:55:28,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:31,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:33,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:36,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:55:36,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:55:39,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:41,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:44,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:49,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:51,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:55:51,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:55:51,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:55:54,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:55:58,843 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:56:00,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:01,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:05,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:06,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:56:06,018 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:56:10,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:11,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:13,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:56:13,862 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:56:13,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:56:13,863 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:56:14,228 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:56:15,228 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:56:15,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:21,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:56:21,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:56:21,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:21,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:26,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:28,845 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:56:31,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:32,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:36,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:56:36,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:56:37,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:41,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:42,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:48,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:51,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:56:51,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:56:51,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:56:54,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:56:58,847 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:57:00,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:01,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:05,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:06,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:57:06,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:57:11,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:11,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:16,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:21,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:57:21,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:57:21,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:22,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:27,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:28,849 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:57:31,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:32,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:36,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:57:36,020 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:57:38,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:41,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:44,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:50,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:51,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:57:51,020 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:57:51,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:57:56,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:57:58,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:58:01,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:01,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:06,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:58:06,020 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:58:07,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:11,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:12,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:16,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 08:58:16,779 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 08:58:16,779 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 08:58:16,780 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 08:58:17,288 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 08:58:17,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:19,289 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 08:58:21,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:58:21,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:58:21,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:23,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:28,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:28,855 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:58:31,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:33,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:36,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:58:36,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:58:39,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:41,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:45,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:50,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:51,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:58:51,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:58:51,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:58:56,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:58:58,863 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:59:01,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:01,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:06,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:59:06,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:59:07,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:11,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:12,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:18,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:21,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:59:21,023 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:59:21,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:24,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:28,860 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 08:59:29,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:31,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:35,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:36,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:59:36,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:59:41,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:41,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:47,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:51,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 08:59:51,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 08:59:51,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 08:59:53,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:58,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 08:59:58,866 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:00:01,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:03,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:06,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:00:06,024 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:00:08,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:11,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:13,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:18,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:00:18,488 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:00:18,489 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:00:18,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:18,490 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:00:19,341 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:00:21,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:00:21,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:00:21,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:21,342 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:00:24,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:28,865 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:00:29,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:31,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:35,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:36,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:00:36,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:00:41,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:41,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:47,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:51,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:00:51,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:00:51,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:00:53,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:58,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:00:58,868 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:01:01,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:04,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:06,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:01:06,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:01:10,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:11,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:15,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:21,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:01:21,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:01:21,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:21,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:26,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:28,870 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:01:31,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:31,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:36,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:01:36,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:01:37,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:41,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:42,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:47,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:51,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:01:51,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:01:51,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:01:53,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:58,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:01:58,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:02:01,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:03,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:06,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:02:06,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:02:08,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:11,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:14,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:19,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:20,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:02:20,375 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:02:20,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:02:20,376 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:02:20,401 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:02:21,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:02:21,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:02:21,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:21,401 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:02:25,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:28,875 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:02:30,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:31,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:36,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:02:36,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:02:36,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:41,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:41,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:46,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:51,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:02:51,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:02:51,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:02:52,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:57,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:02:58,878 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:03:01,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:02,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:06,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:03:06,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:03:08,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:11,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:13,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:18,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:21,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:03:21,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:03:21,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:23,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:28,886 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:03:28,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:31,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:34,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:36,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:03:36,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:03:40,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:41,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:45,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:51,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:03:51,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:03:51,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:03:51,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:56,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:03:58,882 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:04:01,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:01,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:06,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:04:06,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:04:07,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:11,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:12,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:17,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:22,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:04:22,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:04:22,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:23,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:23,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:04:23,809 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:04:23,809 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:04:23,811 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:04:24,467 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:04:25,467 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:04:28,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:28,885 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:04:32,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:33,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:37,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:04:37,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:04:39,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:42,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:44,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:50,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:52,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:04:52,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:04:52,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:04:55,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:04:58,889 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:05:00,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:02,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:06,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:07,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:05:07,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:05:12,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:12,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:17,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:22,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:22,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:05:22,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:05:23,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:28,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:28,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:05:32,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:33,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:37,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:05:37,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:05:39,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:42,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:44,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:50,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:52,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:05:52,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:05:52,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:05:56,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:05:58,892 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:06:01,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:02,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:07,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:06:07,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:06:07,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:12,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:12,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:17,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:22,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:22,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:06:22,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:06:23,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:25,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:06:25,809 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:06:25,810 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:06:25,810 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:06:26,522 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:06:27,523 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:06:28,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:28,897 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:06:32,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:34,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:37,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:06:37,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:06:40,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:42,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:45,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:50,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:52,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:06:52,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:06:52,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:06:56,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:06:58,898 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:07:01,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:02,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:07,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:07:07,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:07:07,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:12,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:12,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:18,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:22,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:22,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:07:22,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:07:24,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:28,900 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:07:29,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:32,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:35,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:37,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:07:37,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:07:41,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:42,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:46,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:52,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:07:52,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:07:52,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:07:52,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:57,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:07:58,907 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:08:02,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:02,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:07,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:08:07,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:08:08,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:12,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:13,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:18,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:22,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:22,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:08:22,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:08:23,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:27,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:08:27,698 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:08:27,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:08:27,699 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:08:28,582 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:08:28,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:08:28,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:29,582 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:08:32,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:34,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:37,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:08:37,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:08:40,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:42,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:45,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:51,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:52,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:08:52,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:08:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:08:57,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:08:58,909 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:09:02,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:02,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:07,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:09:07,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:09:08,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:12,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:13,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:18,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:22,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:09:22,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:09:22,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:24,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:28,912 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:09:29,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:32,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:35,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:37,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:09:37,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:09:41,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:42,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:47,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:52,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:09:52,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:09:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:09:53,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:58,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:09:58,913 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:10:02,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:03,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:07,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:10:07,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:10:08,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:12,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:14,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:19,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:22,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:10:22,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:10:22,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:25,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:28,915 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:10:29,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:10:29,514 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:10:29,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:10:29,515 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:10:29,637 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:10:30,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:31,638 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:10:32,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:36,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:37,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:10:37,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:10:42,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:42,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:47,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:52,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:10:52,038 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:10:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:10:53,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:58,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:10:58,917 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:11:02,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:03,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:07,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:11:07,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:11:09,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:12,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:14,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:20,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:22,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:22,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:11:22,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:11:26,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:28,919 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:11:31,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:32,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:37,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:11:37,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:11:37,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:42,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:42,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:48,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:52,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:11:52,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:11:52,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:11:54,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:11:58,922 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:11:59,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:02,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:05,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:07,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:12:07,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:12:11,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:12,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:16,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:22,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:22,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:12:22,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:12:22,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:27,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:28,929 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:12:32,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:32,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:12:32,792 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:12:32,793 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:12:32,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:32,794 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:12:33,695 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:12:34,695 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:12:37,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:12:37,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:12:38,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:42,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:43,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:48,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:52,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:12:52,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:12:52,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:12:54,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:12:58,927 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:12:59,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:02,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:05,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:07,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:13:07,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:13:11,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:12,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:17,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:22,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:22,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:13:22,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:13:23,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:28,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:28,931 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:13:32,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:34,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:37,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:13:37,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:13:40,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:42,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:46,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:51,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:52,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:13:52,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:13:52,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:13:57,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:13:58,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:14:02,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:02,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:07,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:14:07,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:14:08,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:12,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:13,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:18,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:22,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:22,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:14:22,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:14:24,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:28,934 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:14:29,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:32,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:34,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:14:34,686 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:14:34,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:14:34,687 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:14:34,739 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:14:35,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:36,740 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:14:37,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:14:37,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:14:41,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:42,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:46,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:52,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:14:52,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:14:52,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:14:52,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:57,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:14:58,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:15:02,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:02,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:07,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:15:07,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:15:08,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:12,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:13,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:18,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:22,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:22,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:15:22,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:15:23,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:28,940 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:15:28,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:32,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:34,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:37,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:15:37,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:15:40,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:42,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:45,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:51,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:52,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:15:52,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:15:52,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:15:57,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:15:58,942 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:16:02,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:02,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:07,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:16:07,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:16:08,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:12,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:13,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:19,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:22,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:22,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:16:22,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:16:25,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:28,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:16:30,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:32,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:36,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:16:36,451 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:16:36,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:16:36,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:36,452 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:16:36,807 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:16:36,808 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:16:37,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:16:37,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:16:41,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:42,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:47,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:52,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:16:52,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:16:52,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:16:53,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:58,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:16:58,951 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:17:02,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:03,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:07,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:17:07,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:17:09,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:12,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:15,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:20,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:22,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:22,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:17:22,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:17:26,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:28,949 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:17:31,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:32,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:37,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:17:37,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:17:37,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:42,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:42,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:47,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:52,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:17:52,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:17:52,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:17:53,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:58,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:17:58,954 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:18:02,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:04,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:07,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:18:07,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:18:09,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:12,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:15,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:20,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:22,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:22,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:18:22,067 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:18:25,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:28,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:18:30,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:32,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:36,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:37,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:18:37,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:18:39,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:18:39,634 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:18:39,634 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:18:39,635 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:18:39,858 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:18:40,860 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:18:41,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:42,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:47,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:52,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:18:52,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:18:52,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:18:53,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:58,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:18:58,959 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:19:02,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:03,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:07,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:19:07,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:19:09,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:12,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:14,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:20,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:22,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:22,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:19:22,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:19:26,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:28,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:19:31,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:32,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:37,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:19:37,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:19:37,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:42,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:42,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:47,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:52,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:19:52,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:19:52,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:19:53,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:58,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:19:58,963 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:20:02,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:04,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:07,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:20:07,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:20:10,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:12,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:16,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:22,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:22,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:20:22,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:20:22,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:27,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:28,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:20:32,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:32,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:37,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:20:37,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:20:38,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:41,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:20:41,255 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:20:41,255 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:20:41,256 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:20:41,919 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:20:42,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:42,920 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:20:43,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:48,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:52,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:20:52,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:20:52,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:20:54,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:20:58,968 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:20:59,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:02,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:05,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:07,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:21:07,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:21:11,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:12,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:17,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:22,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:22,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:21:22,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:21:23,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:28,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:28,970 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:21:32,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:33,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:37,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:21:37,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:21:39,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:42,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:44,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:50,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:52,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:21:52,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:21:52,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:21:56,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:21:58,976 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:22:01,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:02,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:07,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:22:07,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:22:07,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:12,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:12,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:18,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:22,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:22,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:22:22,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:22:23,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:28,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:28,975 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:22:32,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:33,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:37,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:22:37,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:22:39,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:42,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:42,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:22:42,974 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:22:42,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:22:42,977 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:22:42,978 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:22:43,977 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:22:44,978 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:22:45,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:50,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:52,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:22:52,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:22:52,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:22:56,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:22:58,977 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:23:01,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:02,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:07,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:23:07,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:23:07,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:12,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:12,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:17,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:22,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:22,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:23:22,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:23:23,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:28,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:28,980 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:23:32,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:33,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:37,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:23:37,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:23:39,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:42,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:44,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:49,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:52,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:23:52,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:23:52,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:23:55,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:23:58,983 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:24:01,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:02,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:06,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:07,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:24:07,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:24:12,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:12,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:17,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:22,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:22,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:24:22,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:24:23,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:28,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:28,984 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:24:32,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:33,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:37,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:24:37,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:24:39,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:42,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:44,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:45,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:24:45,984 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:24:45,985 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:24:45,985 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:24:46,037 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:24:47,038 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:24:50,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:52,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:24:52,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:24:52,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:24:55,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:24:58,986 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:25:01,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:02,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:07,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:25:07,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:25:07,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:12,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:12,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:18,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:22,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:25:22,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:25:22,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:24,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:28,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:25:29,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:32,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:34,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:37,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:25:37,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:25:39,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:42,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:44,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:50,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:52,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:25:52,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:25:52,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:25:55,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:25:58,992 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:26:01,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:02,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:06,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:07,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:26:07,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:26:11,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:12,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:17,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:22,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:26:22,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:26:22,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:23,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:28,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:28,997 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:26:32,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:33,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:37,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:26:37,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:26:39,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:42,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:44,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:47,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:26:47,648 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:26:47,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:26:47,649 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:26:48,085 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:26:49,086 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:26:50,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:26:52,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:26:52,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:26:55,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:26:58,996 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:27:01,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:02,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:06,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:07,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:27:07,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:27:11,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:12,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:17,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:22,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:27:22,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:27:22,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:23,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:28,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:29,000 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:27:32,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:34,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:37,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:27:37,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:27:40,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:42,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:46,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:51,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:52,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:27:52,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:27:52,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:27:57,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:27:59,001 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:28:02,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:02,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:07,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:28:07,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:28:08,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:12,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:13,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:19,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:22,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:28:22,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:28:22,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:25,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:29,003 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:28:30,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:32,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:35,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:37,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:28:37,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:28:40,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:42,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:46,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:49,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:28:49,086 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:28:49,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:28:49,087 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:28:49,130 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:28:51,132 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:28:51,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:52,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:28:52,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:28:52,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:28:57,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:28:59,006 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:29:02,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:03,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:07,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:29:07,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:29:09,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:12,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:14,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:19,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:22,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:29:22,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:29:22,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:24,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:29,008 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:29:30,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:32,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:35,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:37,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:29:37,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:29:41,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:42,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:47,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:52,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:29:52,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:29:52,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:29:53,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:58,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:29:59,011 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:30:02,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:04,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:07,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:30:07,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:30:09,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:12,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:15,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:21,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:22,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:30:22,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:30:22,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:26,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:29,013 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:30:32,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:32,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:37,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:30:37,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:30:38,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:42,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:43,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:48,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:52,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:30:52,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:30:52,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:30:52,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:30:52,296 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:30:52,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:30:52,298 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:30:53,174 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:30:53,174 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:30:54,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:30:59,019 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:30:59,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:02,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:05,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:07,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:31:07,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:31:10,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:12,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:16,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:21,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:22,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:31:22,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:31:22,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:26,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:29,017 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:31:31,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:32,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:37,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:31:37,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:31:37,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:42,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:42,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:47,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:52,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:31:52,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:31:52,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:31:53,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:58,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:31:59,021 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:32:02,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:03,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:07,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:32:07,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:32:09,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:12,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:14,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:19,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:22,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:32:22,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:32:22,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:25,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:29,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:32:30,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:32,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:35,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:37,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:32:37,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:32:41,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:42,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:46,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:52,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:32:52,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:32:52,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:32:52,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:53,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:32:53,904 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:32:53,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:32:53,905 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:32:54,224 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:32:55,225 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:32:57,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:32:59,025 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:33:02,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:02,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:07,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:33:07,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:33:08,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:12,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:13,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:19,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:22,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:33:22,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:33:22,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:24,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:29,028 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:33:29,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:32,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:35,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:37,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:33:37,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:33:41,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:42,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:46,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:51,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:52,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:33:52,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:33:52,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:33:57,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:33:59,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:34:02,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:02,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:07,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:34:07,085 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:34:08,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:12,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:13,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:19,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:22,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:34:22,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:34:22,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:25,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:29,033 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:34:30,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:32,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:36,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:37,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:34:37,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:34:41,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:42,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:46,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:52,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:34:52,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:34:52,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:34:52,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:55,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:34:55,613 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:34:55,614 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:34:55,614 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:34:56,281 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:34:57,281 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:34:57,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:34:59,036 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:35:02,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:02,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:07,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:35:07,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:35:08,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:12,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:13,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:19,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:22,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:35:22,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:35:22,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:25,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:29,042 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:35:30,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:32,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:36,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:37,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:35:37,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:35:42,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:42,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:48,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:52,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:35:52,087 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:35:52,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:35:54,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:35:59,041 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:35:59,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:02,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:05,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:07,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:36:07,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:36:11,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:12,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:16,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:21,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:22,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:36:22,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:36:22,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:27,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:29,044 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:36:32,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:32,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:37,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:36:37,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:36:38,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:42,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:43,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:49,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:52,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:36:52,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:36:52,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:36:55,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:36:58,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:36:58,743 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:36:58,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:36:58,745 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:36:59,045 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:36:59,327 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:37:00,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:01,328 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:37:02,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:06,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:07,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:37:07,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:37:11,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:12,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:17,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:22,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:37:22,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:37:22,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:23,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:28,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:29,047 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:37:32,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:34,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:37,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:37:37,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:37:40,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:42,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:45,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:50,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:52,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:37:52,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:37:52,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:37:56,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:37:59,050 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:38:01,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:02,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:07,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:38:07,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:38:07,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:12,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:12,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:18,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:22,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:38:22,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:38:22,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:24,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:29,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:38:30,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:32,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:35,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:37,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:38:37,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:38:41,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:42,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:47,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:52,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:38:52,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:38:52,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:38:53,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:58,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:38:59,054 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:39:00,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:39:00,597 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:39:00,598 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:39:00,599 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:39:01,384 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:39:01,384 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:39:02,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:03,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:08,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:39:08,021 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:39:09,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:09,389 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:39:12,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:15,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:19,394 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:39:20,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:22,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:23,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:39:23,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:39:25,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:29,056 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:39:31,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:32,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:37,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:38,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:39:38,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:39:42,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:42,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:48,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:52,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:39:53,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:39:53,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:39:53,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:59,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:39:59,063 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:40:02,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:04,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:08,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:40:08,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:40:09,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:12,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:15,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:21,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:22,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:23,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:40:23,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:40:26,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:29,061 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:40:32,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:32,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:38,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:40:38,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:40:38,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:42,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:44,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:49,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:52,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:40:53,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:40:53,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:40:54,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:40:59,065 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:41:00,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:02,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:06,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:08,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:41:08,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:41:11,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:12,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:17,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:18,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:41:18,798 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:41:18,798 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:41:18,799 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:41:19,463 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:41:21,464 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:41:22,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:22,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:23,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:41:23,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:41:28,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:29,066 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:41:32,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:34,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:38,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:41:38,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:41:39,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:42,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:45,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:50,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:52,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:41:53,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:41:53,028 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:41:55,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:41:59,070 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:42:01,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:02,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:07,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:08,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:42:08,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:42:12,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:12,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:18,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:22,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:23,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:42:23,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:42:23,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:29,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:29,073 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:42:32,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:35,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:38,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:42:38,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:42:40,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:42,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:46,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:51,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:52,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:42:53,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:42:53,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:42:56,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:42:59,074 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:43:02,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:02,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:08,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:43:08,029 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:43:08,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:12,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:14,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:19,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:20,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:43:20,385 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:43:20,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:43:20,386 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:43:20,525 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:43:21,526 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:43:23,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:23,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:43:23,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:43:24,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:29,077 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:43:30,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:33,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:35,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:38,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:43:38,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:43:40,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:43,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:46,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:52,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:53,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:43:53,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:43:53,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:43:57,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:43:59,080 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:44:03,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:03,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:08,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:44:08,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:44:08,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:13,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:14,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:19,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:23,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:44:23,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:44:23,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:24,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:29,082 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:44:30,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:33,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:35,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:38,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:44:38,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:44:40,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:43,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:46,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:51,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:53,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:44:53,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:44:53,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:44:56,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:44:59,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:45:02,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:03,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:07,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:08,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:45:08,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:45:12,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:13,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:18,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:23,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:45:23,351 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:45:23,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:23,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:23,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:45:23,545 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:45:23,546 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:45:23,547 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:45:23,604 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:45:25,605 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:45:29,087 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:45:29,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:33,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:34,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:38,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:45:38,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:45:39,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:43,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:45,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:50,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:53,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:45:53,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:45:53,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:45:55,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:45:59,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:46:01,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:03,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:06,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:08,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:46:08,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:46:11,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:13,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:17,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:23,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:46:23,352 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:46:23,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:23,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:29,091 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:46:29,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:33,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:34,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:38,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:46:38,353 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:46:39,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:43,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:45,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:50,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:53,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:46:53,353 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:46:53,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:46:55,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:46:59,094 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:47:01,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:03,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:06,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:08,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:47:08,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:47:11,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:13,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:17,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:22,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:23,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:47:23,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:47:23,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:25,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:47:25,244 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:47:25,244 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:47:25,245 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:47:25,669 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:47:27,670 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:47:28,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:29,096 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:47:33,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:33,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:38,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:47:38,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:47:38,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:43,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:44,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:49,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:53,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:47:53,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:47:53,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:47:54,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:47:59,098 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:48:00,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:03,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:05,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:08,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:48:08,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:48:10,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:13,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:16,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:22,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:23,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:48:23,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:48:23,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:27,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:29,101 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:48:33,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:33,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:38,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:48:38,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:48:38,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:43,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:44,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:49,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:53,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:48:53,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:48:53,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:48:54,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:48:59,103 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:49:00,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:03,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:05,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:08,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:49:08,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:49:10,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:13,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:16,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:22,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:23,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:49:23,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:49:23,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:27,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:49:27,132 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:49:27,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:49:27,133 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:49:27,751 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:49:28,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:29,110 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:49:29,753 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:49:33,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:33,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:38,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:49:38,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:49:38,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:43,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:44,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:50,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:53,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:49:53,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:49:53,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:49:55,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:49:59,107 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:50:01,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:03,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:06,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:08,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:50:08,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:50:11,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:13,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:17,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:22,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:23,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:50:23,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:50:23,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:27,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:29,111 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:50:33,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:33,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:38,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:50:38,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:50:38,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:43,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:44,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:49,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:53,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:50:53,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:50:53,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:50:54,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:50:59,112 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:51:00,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:03,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:05,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:08,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:51:08,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:51:10,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:13,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:16,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:21,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:23,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:51:23,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:51:23,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:26,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:29,114 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:51:30,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:51:30,548 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:51:30,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:51:30,549 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:51:30,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:51:31,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:31,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:51:33,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:37,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:38,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:51:38,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:51:42,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:43,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:48,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:53,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:51:53,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:51:53,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:51:53,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:51:59,116 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:51:59,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:03,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:04,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:08,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:52:08,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:52:09,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:13,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:15,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:20,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:23,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:52:23,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:52:23,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:25,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:29,119 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:52:31,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:33,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:36,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:38,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:52:38,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:52:41,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:43,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:47,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:52,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:53,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:52:53,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:52:53,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:52:57,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:52:59,122 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:53:03,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:03,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:08,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:53:08,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:53:08,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:13,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:14,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:19,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:23,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:53:23,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:53:23,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:24,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:29,124 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:53:30,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:32,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:53:32,220 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:53:32,220 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:53:32,221 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:53:32,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:53:33,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:33,901 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:53:35,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:38,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:53:38,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:53:40,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:43,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:46,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:52,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:53,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:53:53,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:53:53,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:53:57,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:53:59,130 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:54:03,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:03,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:08,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:54:08,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:54:08,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:13,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:14,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:19,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:23,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:54:23,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:54:23,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:24,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:29,128 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:54:30,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:33,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:35,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:38,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:54:38,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:54:40,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:43,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:46,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:51,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:53,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:54:53,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:54:53,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:54:56,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:54:59,132 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:55:02,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:03,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:07,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:08,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:55:08,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:55:12,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:13,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:18,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:23,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:55:23,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:55:23,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:23,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:29,134 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:55:29,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:33,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:33,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:55:33,936 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:55:33,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:55:33,938 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:55:33,977 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:55:34,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:35,978 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:55:38,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:55:38,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:55:40,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:43,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:46,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:51,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:53,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:55:53,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:55:53,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:55:56,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:55:59,137 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:56:02,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:03,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:07,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:08,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:56:08,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:56:12,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:13,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:18,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:23,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:56:23,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:56:23,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:23,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:29,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:56:29,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:33,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:34,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:38,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:56:38,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:56:39,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:43,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:45,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:50,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:53,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:56:53,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:56:53,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:56:56,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:56:59,141 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:57:01,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:03,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:06,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:08,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:57:08,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:57:12,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:13,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:17,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:22,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:23,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:57:23,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:57:23,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:28,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:29,143 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:57:33,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:34,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:37,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:57:37,099 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:57:37,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:57:37,100 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:57:38,037 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:57:38,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:57:38,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:57:39,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:40,038 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:57:43,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:45,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:50,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:53,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:57:53,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:57:53,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:57:56,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:57:59,146 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:58:01,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:03,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:07,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:08,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:58:08,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:58:13,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:13,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:19,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:23,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:58:23,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:58:23,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:25,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:29,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:58:30,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:33,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:36,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:38,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:58:38,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:58:42,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:43,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:47,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:53,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:58:53,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:58:53,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:58:53,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:58:59,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:58:59,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:03,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:04,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:08,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:59:08,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:59:10,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:13,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:15,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:21,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:23,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:59:23,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:59:23,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:27,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:29,154 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 09:59:32,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:33,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:38,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:59:38,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:59:38,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:38,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 09:59:38,830 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 09:59:38,830 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 09:59:38,831 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 09:59:39,096 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 09:59:40,097 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 09:59:43,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:44,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:49,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:53,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 09:59:53,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 09:59:53,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 09:59:55,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 09:59:59,155 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:00:00,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:03,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:05,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:08,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:00:08,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:00:11,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:13,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:17,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:22,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:23,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:00:23,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:00:23,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:27,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:29,157 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:00:32,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:33,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:38,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:00:38,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:00:38,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:43,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:44,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:50,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:53,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:00:53,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:00:53,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:00:56,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:00:59,160 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:01:01,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:03,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:07,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:08,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:01:08,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:01:13,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:13,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:18,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:23,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:01:23,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:01:23,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:24,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:29,162 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:01:29,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:33,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:34,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:38,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:01:38,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:01:40,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:01:40,435 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:01:40,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:01:40,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:40,437 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:01:41,160 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:01:42,161 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:01:43,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:45,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:50,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:53,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:01:53,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:01:53,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:01:56,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:01:59,164 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:02:02,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:03,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:07,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:08,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:02:08,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:02:13,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:13,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:19,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:23,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:02:23,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:02:23,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:25,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:29,167 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:02:30,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:33,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:36,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:38,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:02:38,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:02:41,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:43,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:47,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:52,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:53,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:02:53,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:02:53,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:02:58,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:02:59,170 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:03:03,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:04,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:08,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:03:08,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:03:10,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:13,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:15,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:21,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:23,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:03:23,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:03:23,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:27,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:29,176 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:03:32,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:33,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:38,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:03:38,380 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:03:38,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:43,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:43,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:43,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:03:43,680 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:03:43,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:03:43,681 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:03:44,232 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:03:46,234 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:03:49,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:53,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:03:53,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:03:53,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:03:55,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:03:59,175 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:04:01,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:03,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:06,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:08,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:04:08,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:04:12,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:13,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:18,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:23,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:04:23,381 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:04:23,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:24,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:29,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:04:29,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:33,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:35,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:38,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:04:38,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:04:40,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:43,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:46,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:51,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:53,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:04:53,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:04:53,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:04:57,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:04:59,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:05:02,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:03,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:08,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:05:08,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:05:08,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:13,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:14,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:19,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:23,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:05:23,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:05:23,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:24,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:29,182 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:05:29,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:33,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:35,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:38,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:05:38,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:05:40,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:43,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:45,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:05:45,212 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:05:45,213 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:05:45,213 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:05:45,285 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:05:46,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:48,286 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:05:51,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:53,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:05:53,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:05:53,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:05:57,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:05:59,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:06:02,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:03,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:08,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:06:08,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:06:08,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:13,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:14,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:19,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:23,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:06:23,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:06:23,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:25,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:29,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:06:30,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:33,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:36,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:38,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:06:38,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:06:42,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:43,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:48,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:53,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:06:53,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:06:53,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:06:53,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:06:59,188 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:06:59,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:03,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:04,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:08,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:07:08,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:07:10,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:13,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:15,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:21,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:23,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:07:23,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:07:23,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:26,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:29,190 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:07:31,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:33,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:37,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:38,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:07:38,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:07:43,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:43,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:46,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:07:46,839 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:07:46,839 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:07:46,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:07:47,330 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:07:48,330 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:07:48,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:53,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:07:53,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:07:53,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:07:54,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:07:59,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:07:59,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:03,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:05,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:08,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:08:08,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:08:10,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:13,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:15,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:21,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:23,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:08:23,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:08:23,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:27,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:29,195 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:08:32,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:33,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:38,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:08:38,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:08:38,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:43,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:43,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:49,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:53,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:08:53,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:08:53,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:08:55,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:08:59,198 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:09:00,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:03,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:06,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:08,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:09:08,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:09:12,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:14,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:18,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:23,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:09:23,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:09:24,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:24,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:29,199 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:09:29,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:34,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:34,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:38,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:09:38,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:09:40,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:44,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:46,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:49,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:09:49,935 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:09:49,935 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:09:49,936 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:09:50,388 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:09:51,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:52,390 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:09:53,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:09:53,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:09:54,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:09:57,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:09:59,202 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:10:02,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:04,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:08,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:10:08,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:10:08,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:14,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:14,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:20,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:23,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:10:23,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:10:24,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:26,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:29,205 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:10:31,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:34,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:36,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:38,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:10:38,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:10:42,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:44,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:48,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:53,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:10:53,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:10:54,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:10:54,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:10:59,207 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:10:59,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:04,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:05,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:08,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:11:08,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:11:11,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:14,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:16,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:21,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:23,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:11:23,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:11:24,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:27,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:29,210 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:11:33,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:34,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:38,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:11:38,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:11:39,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:44,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:44,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:50,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:51,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:11:51,916 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:11:51,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:11:51,918 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:11:52,452 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:11:53,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:11:53,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:11:54,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:11:54,454 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:11:55,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:11:59,212 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:12:00,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:04,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:05,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:08,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:12:08,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:12:11,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:14,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:16,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:22,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:23,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:12:23,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:12:24,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:28,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:29,218 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:12:34,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:34,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:38,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:12:38,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:12:40,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:44,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:45,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:50,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:53,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:12:53,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:12:54,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:12:55,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:12:59,217 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:13:00,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:04,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:06,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:08,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:13:08,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:13:12,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:14,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:17,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:23,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:13:23,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:13:23,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:24,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:28,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:29,220 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:13:34,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:34,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:38,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:13:38,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:13:40,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:44,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:45,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:50,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:53,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:13:53,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:13:53,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:13:53,540 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:13:53,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:13:53,541 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:13:54,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:13:54,504 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:13:56,505 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:13:56,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:13:59,222 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:14:01,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:04,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:07,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:08,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:14:08,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:14:12,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:14,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:18,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:23,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:14:23,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:14:24,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:24,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:29,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:14:29,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:34,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:35,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:38,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:14:38,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:14:41,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:44,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:46,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:51,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:53,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:14:53,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:14:54,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:14:57,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:14:59,226 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:15:02,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:04,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:08,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:15:08,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:15:08,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:13,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:14,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:18,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:23,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:15:23,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:15:24,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:24,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:29,228 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:15:29,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:34,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:34,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:38,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:15:38,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:15:40,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:44,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:45,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:50,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:53,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:15:53,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:15:54,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:15:56,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:15:56,693 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:15:56,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:15:56,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:15:56,694 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:15:57,556 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:15:58,557 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:15:59,231 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:16:01,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:04,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:06,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:08,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:16:08,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:16:12,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:14,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:17,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:23,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:16:23,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:16:23,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:24,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:28,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:29,233 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:16:33,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:34,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:38,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:16:38,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:16:39,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:44,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:44,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:49,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:53,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:16:53,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:16:54,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:16:55,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:16:59,245 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:17:00,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:04,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:06,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:08,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:17:08,411 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:17:12,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:14,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:17,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:23,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:17:23,411 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:17:23,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:24,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:28,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:29,238 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:17:34,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:34,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:38,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:17:38,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:17:40,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:44,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:45,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:51,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:53,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:17:53,415 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:17:54,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:17:57,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:17:58,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:17:58,451 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:17:58,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:17:58,454 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:17:58,615 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:17:59,241 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:18:00,615 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:18:02,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:04,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:08,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:18:08,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:18:08,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:13,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:14,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:18,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:23,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:18:23,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:18:24,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:24,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:29,243 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:18:30,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:34,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:35,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:38,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:18:38,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:18:40,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:44,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:45,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:50,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:53,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:18:53,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:18:54,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:18:55,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:18:59,246 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:19:01,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:04,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:07,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:08,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:19:08,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:19:12,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:14,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:18,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:23,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:19:23,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:19:24,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:24,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:29,248 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:19:30,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:34,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:35,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:38,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:19:38,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:19:41,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:44,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:47,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:52,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:53,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:19:53,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:19:54,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:19:58,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:19:59,250 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:20:00,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:20:00,063 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:20:00,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:20:00,064 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:20:00,670 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:20:02,671 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:20:04,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:04,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:08,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:20:08,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:20:09,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:14,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:14,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:21,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:23,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:20:23,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:20:24,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:26,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:29,254 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:20:32,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:34,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:38,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:38,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:20:38,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:20:44,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:44,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:50,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:53,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:20:53,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:20:54,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:20:55,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:20:59,256 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:21:01,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:04,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:07,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:08,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:21:08,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:21:13,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:14,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:18,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:23,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:23,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:21:23,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:21:24,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:28,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:29,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:21:34,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:34,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:38,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:21:38,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:21:39,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:44,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:45,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:51,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:53,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:21:53,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:21:54,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:21:56,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:21:59,264 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:22:01,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:01,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:22:01,624 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:22:01,624 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:22:01,625 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:22:01,726 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:22:04,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:04,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:22:07,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:08,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:22:08,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:22:12,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:14,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:17,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:23,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:23,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:22:23,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:22:24,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:29,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:29,263 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:22:34,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:34,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:38,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:22:38,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:22:39,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:44,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:45,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:50,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:53,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:22:53,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:22:54,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:22:55,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:22:59,267 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:23:01,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:04,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:07,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:08,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:23:08,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:23:12,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:14,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:18,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:23,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:23,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:23:23,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:23:24,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:28,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:29,267 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:23:34,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:34,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:38,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:23:38,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:23:39,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:44,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:45,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:51,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:53,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:23:53,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:23:54,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:23:57,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:23:59,270 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:24:02,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:04,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:04,784 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:24:04,786 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:24:04,786 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:24:04,787 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:24:05,782 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:24:06,784 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:24:08,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:08,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:24:08,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:24:14,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:14,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:19,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:23,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:24:23,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:24:24,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:24,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:29,272 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:24:30,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:34,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:36,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:38,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:24:38,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:24:41,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:44,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:47,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:53,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:53,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:24:53,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:24:54,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:24:59,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:24:59,275 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:25:04,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:04,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:08,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:25:08,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:25:09,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:14,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:15,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:20,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:23,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:25:23,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:25:24,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:25,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:29,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:25:31,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:34,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:37,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:38,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:25:38,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:25:42,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:44,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:48,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:53,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:25:53,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:25:53,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:54,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:25:59,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:25:59,279 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:26:04,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:05,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:06,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:26:06,295 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:26:06,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:26:06,296 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:26:06,838 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:26:08,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:26:08,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:26:08,839 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:26:10,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:14,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:16,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:21,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:23,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:26:23,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:26:24,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:27,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:29,347 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:26:32,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:34,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:38,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:38,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:26:38,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:26:44,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:44,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:49,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:53,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:26:53,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:26:54,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:26:54,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:26:59,346 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:27:00,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:04,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:06,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:08,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:27:08,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:27:12,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:14,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:17,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:23,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:23,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:27:23,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:27:24,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:29,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:29,349 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:27:34,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:35,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:38,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:27:38,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:27:40,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:44,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:46,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:51,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:53,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:27:53,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:27:54,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:27:57,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:27:59,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:28:03,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:04,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:07,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:28:07,879 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:28:07,880 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:28:07,880 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:28:07,888 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:28:08,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:28:08,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:28:08,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:09,891 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:28:14,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:14,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:20,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:23,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:28:23,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:28:24,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:25,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:29,354 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:28:31,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:34,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:37,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:38,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:28:38,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:28:43,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:44,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:49,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:53,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:28:53,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:28:54,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:28:54,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:28:59,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:29:00,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:04,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:06,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:08,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:29:08,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:29:12,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:14,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:18,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:23,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:23,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:29:23,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:29:24,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:28,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:29,358 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:29:34,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:34,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:38,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:29:38,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:29:39,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:44,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:45,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:50,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:53,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:29:53,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:29:54,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:29:55,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:29:59,360 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:30:01,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:04,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:06,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:08,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:30:08,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:30:11,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:30:11,019 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:30:11,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:30:11,020 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:30:11,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:11,948 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:30:11,949 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:30:14,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:17,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:23,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:23,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:30:23,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:30:24,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:28,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:29,362 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:30:34,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:34,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:38,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:30:38,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:30:39,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:44,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:45,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:51,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:53,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:30:53,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:30:54,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:30:56,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:30:59,369 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:31:01,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:04,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:06,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:08,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:31:08,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:31:12,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:14,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:17,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:22,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:23,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:31:23,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:31:24,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:28,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:29,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:31:33,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:34,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:38,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:31:38,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:31:38,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:44,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:44,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:50,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:53,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:31:53,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:31:54,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:31:55,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:31:59,372 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:32:01,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:04,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:06,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:08,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:32:08,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:32:12,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:12,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:32:12,566 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:32:12,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:32:12,567 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:32:13,011 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:32:14,012 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:32:14,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:18,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:23,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:32:23,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:32:23,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:24,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:29,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:29,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:32:34,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:35,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:38,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:32:38,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:32:40,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:44,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:46,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:51,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:53,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:32:53,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:32:54,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:32:57,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:32:59,375 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:33:02,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:04,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:07,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:08,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:33:08,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:33:13,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:14,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:19,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:23,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:33:23,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:33:24,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:24,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:29,377 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:33:30,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:34,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:36,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:38,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:33:38,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:33:42,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:44,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:48,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:53,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:53,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:33:53,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:33:54,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:33:58,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:33:59,379 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:34:03,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:04,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:08,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:34:08,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:34:08,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:14,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:14,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:14,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:34:14,222 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:34:14,222 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:34:14,223 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:34:15,073 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:34:16,074 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:34:20,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:23,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:34:23,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:34:24,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:25,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:29,382 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:34:31,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:34,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:36,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:38,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:34:38,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:34:41,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:44,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:47,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:53,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:53,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:34:53,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:34:54,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:34:59,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:34:59,384 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:35:04,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:04,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:08,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:35:08,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:35:09,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:14,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:15,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:21,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:23,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:35:23,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:35:24,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:26,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:29,389 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:35:32,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:34,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:37,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:38,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:35:38,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:35:43,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:44,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:48,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:53,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:53,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:35:53,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:35:54,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:35:59,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:35:59,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:36:04,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:05,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:08,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:36:08,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:36:10,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:14,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:16,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:17,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:36:17,530 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:36:17,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:36:17,533 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:36:18,128 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:36:18,128 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:36:22,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:23,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:36:23,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:36:24,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:27,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:29,391 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:36:32,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:34,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:38,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:38,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:36:38,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:36:43,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:44,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:49,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:53,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:36:53,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:36:54,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:36:54,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:36:59,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:37:00,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:04,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:06,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:08,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:37:08,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:37:12,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:14,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:17,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:22,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:23,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:37:23,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:37:24,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:28,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:29,395 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:37:34,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:34,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:38,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:37:38,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:37:39,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:44,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:45,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:50,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:53,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:37:53,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:37:54,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:37:55,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:37:59,397 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:38:01,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:04,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:06,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:08,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:38:08,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:38:12,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:14,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:18,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:19,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:38:19,034 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:38:19,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:38:19,035 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:38:19,175 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:38:20,177 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:38:23,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:23,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:38:23,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:38:24,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:29,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:29,400 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:38:34,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:34,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:38,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:38:38,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:38:39,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:44,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:45,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:50,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:53,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:38:53,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:38:54,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:38:55,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:38:59,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:39:01,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:04,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:07,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:08,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:39:08,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:39:12,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:14,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:18,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:23,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:23,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:39:23,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:39:24,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:29,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:29,404 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:39:34,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:34,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:38,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:39:38,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:39:39,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:44,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:45,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:50,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:53,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:39:53,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:39:54,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:39:55,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:39:59,407 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:40:01,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:04,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:07,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:08,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:40:08,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:40:13,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:14,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:19,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:20,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:40:20,688 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:40:20,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:40:20,690 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:40:21,235 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:40:22,237 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:40:23,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:40:23,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:40:24,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:24,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:29,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:40:30,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:34,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:36,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:38,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:40:38,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:40:42,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:44,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:48,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:53,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:53,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:40:53,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:40:54,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:40:59,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:40:59,412 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:41:04,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:04,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:08,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:41:08,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:41:09,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:14,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:15,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:21,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:23,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:41:23,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:41:24,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:27,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:29,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:41:33,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:34,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:38,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:38,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:41:38,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:41:44,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:44,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:49,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:53,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:41:53,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:41:54,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:41:54,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:41:59,417 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:42:00,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:04,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:06,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:08,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:42:08,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:42:12,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:14,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:17,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:23,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:23,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:42:23,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:42:23,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:42:23,763 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:42:23,763 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:42:23,764 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:42:24,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:24,287 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:42:24,287 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:42:28,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:29,419 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:42:34,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:34,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:38,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:42:38,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:42:39,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:44,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:45,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:50,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:53,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:42:53,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:42:54,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:42:55,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:42:59,422 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:43:01,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:04,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:07,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:08,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:43:08,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:43:13,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:14,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:19,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:23,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:43:23,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:43:24,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:24,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:29,424 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:43:30,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:34,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:36,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:38,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:43:38,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:43:42,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:44,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:47,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:53,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:53,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:43:53,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:43:54,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:43:59,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:43:59,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:44:04,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:04,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:08,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:44:08,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:44:09,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:14,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:15,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:21,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:23,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:44:23,466 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:44:24,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:25,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:44:25,179 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:44:25,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:44:25,180 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:44:25,339 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:44:26,340 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:44:27,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:29,428 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:44:32,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:34,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:38,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:38,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:44:38,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:44:44,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:44,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:50,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:53,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:44:53,470 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:44:54,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:44:55,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:44:59,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:45:01,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:04,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:07,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:08,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:45:08,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:45:12,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:14,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:18,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:23,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:23,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:45:23,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:45:24,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:29,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:29,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:45:34,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:35,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:38,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:45:38,471 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:45:40,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:44,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:46,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:52,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:53,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:45:53,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:45:54,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:45:57,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:45:59,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:46:03,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:04,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:08,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:46:08,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:46:08,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:14,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:14,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:19,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:23,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:46:23,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:46:24,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:24,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:26,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:46:26,669 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:46:26,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:46:26,670 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:46:27,383 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:46:28,384 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:46:29,439 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:46:30,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:34,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:36,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:38,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:46:38,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:46:41,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:44,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:47,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:52,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:53,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:46:53,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:46:54,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:46:58,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:46:59,440 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:47:04,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:04,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:08,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:47:08,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:47:09,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:14,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:15,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:21,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:23,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:47:23,473 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:47:24,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:26,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:29,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:47:31,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:34,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:37,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:38,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:47:38,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:47:43,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:44,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:49,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:53,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:47:53,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:47:54,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:47:54,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:47:59,445 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:48:00,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:04,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:06,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:09,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:48:09,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:48:12,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:14,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:18,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:23,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:24,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:24,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:48:24,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:48:29,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:29,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:48:29,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:48:29,895 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:48:29,895 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:48:29,896 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:48:30,444 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:48:30,444 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:48:34,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:35,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:39,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:48:39,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:48:40,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:44,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:46,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:51,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:54,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:48:54,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:48:54,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:48:57,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:48:59,450 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:49:02,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:04,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:08,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:09,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:49:09,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:49:14,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:14,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:19,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:24,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:24,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:49:24,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:49:24,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:29,456 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:49:30,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:34,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:36,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:39,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:49:39,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:49:41,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:44,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:46,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:52,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:54,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:49:54,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:49:54,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:49:57,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:49:59,455 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:50:03,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:04,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:08,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:09,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:50:09,272 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:50:14,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:14,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:20,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:24,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:24,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:50:24,273 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:50:25,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:29,458 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:50:30,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:31,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:50:31,487 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:50:31,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:50:31,488 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:50:31,491 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:50:32,492 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:50:34,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:35,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:39,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:50:39,273 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:50:41,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:44,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:47,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:52,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:54,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:50:54,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:50:54,273 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:50:57,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:50:59,459 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:51:03,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:04,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:08,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:09,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:51:09,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:51:13,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:14,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:18,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:24,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:24,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:24,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:51:24,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:51:29,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:29,461 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:51:34,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:35,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:39,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:51:39,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:51:40,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:44,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:46,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:51,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:54,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:51:54,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:51:54,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:51:56,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:51:59,464 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:52:02,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:04,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:07,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:09,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:52:09,275 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:52:13,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:14,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:18,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:24,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:24,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:24,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:52:24,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:52:29,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:29,466 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:52:33,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:52:33,090 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:52:33,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:52:33,091 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:52:33,547 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:52:34,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:34,548 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:52:35,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:39,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:52:39,275 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:52:40,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:44,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:46,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:52,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:54,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:52:54,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:52:54,275 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:52:58,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:52:59,468 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:53:03,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:04,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:08,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:09,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:53:09,275 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:53:14,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:14,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:19,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:24,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:24,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:53:24,275 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:53:24,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:29,471 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:53:30,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:34,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:36,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:39,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:53:39,276 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:53:42,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:44,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:47,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:52,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:54,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:53:54,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:53:54,276 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:53:58,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:53:59,477 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:54:04,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:04,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:09,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:09,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:54:09,276 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:54:14,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:14,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:20,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:24,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:24,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:54:24,276 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:54:25,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:29,475 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:54:30,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:34,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:34,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:54:34,557 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:54:34,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:54:34,559 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:54:34,601 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:54:35,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:36,603 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:54:39,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:54:39,277 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:54:41,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:44,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:46,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:52,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:54,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:54:54,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:54:54,277 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:54:57,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:54:59,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:55:02,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:04,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:07,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:09,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:55:09,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:55:13,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:14,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:19,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:24,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:24,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:55:24,277 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:55:25,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:29,482 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:55:30,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:34,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:35,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:39,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:55:39,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:55:41,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:44,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:46,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:52,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:54,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:55:54,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:55:54,277 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:55:58,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:55:59,483 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:56:03,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:04,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:09,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:56:09,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:56:09,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:14,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:14,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:19,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:24,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:24,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:56:24,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:56:25,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:29,485 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:56:30,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:34,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:35,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:37,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:56:37,706 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:56:37,706 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:56:37,707 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:56:38,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:56:38,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:56:39,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:56:39,278 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:56:41,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:44,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:46,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:52,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:54,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:56:54,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:56:54,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:56:58,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:56:59,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:57:03,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:04,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:09,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:57:09,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:57:09,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:14,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:14,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:20,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:24,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:24,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:57:24,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:57:26,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:29,490 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:57:31,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:34,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:36,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:39,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:57:39,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:57:42,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:44,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:47,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:52,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:54,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:57:54,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:57:54,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:57:58,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:57:59,492 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:58:04,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:04,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:09,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:58:09,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:58:10,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:14,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:15,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:20,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:24,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:24,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:58:24,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:58:26,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:29,495 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:58:31,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:34,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:36,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:39,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:58:39,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:58:39,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 10:58:39,503 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 10:58:39,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 10:58:39,505 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 10:58:39,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 10:58:40,702 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 10:58:42,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:44,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:47,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:52,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:54,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:58:54,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:58:54,280 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:58:57,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:58:59,501 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:59:03,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:04,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:08,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:09,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:59:09,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:59:14,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:14,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:19,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:24,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:24,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:59:24,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:59:25,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:29,499 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 10:59:30,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:34,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:35,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:39,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:59:39,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:59:41,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:44,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:46,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:52,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:54,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 10:59:54,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 10:59:54,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 10:59:58,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 10:59:59,504 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:00:03,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:04,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:09,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:00:09,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:00:09,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:14,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:14,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:19,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:24,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:24,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:00:24,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:00:25,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:29,504 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:00:30,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:34,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:35,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:39,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:00:39,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:00:40,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:00:40,763 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:00:40,764 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:00:40,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:40,765 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:00:41,744 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:00:42,745 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:00:44,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:46,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:51,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:54,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:00:54,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:00:54,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:00:57,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:00:59,507 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:01:02,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:04,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:07,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:09,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:01:09,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:01:13,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:14,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:18,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:23,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:24,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:24,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:01:24,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:01:28,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:29,509 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:01:33,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:34,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:38,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:39,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:01:39,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:01:44,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:44,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:49,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:54,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:01:54,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:01:54,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:01:55,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:01:59,512 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:02:00,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:04,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:05,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:09,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:02:09,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:02:11,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:14,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:16,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:22,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:24,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:24,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:02:24,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:02:28,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:29,514 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:02:33,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:34,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:39,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:02:39,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:02:39,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:43,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:02:43,938 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:02:43,938 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:02:43,939 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:02:44,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:44,803 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:02:44,803 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:02:44,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:50,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:54,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:02:54,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:02:54,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:02:56,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:02:59,516 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:03:01,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:04,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:06,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:09,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:03:09,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:03:12,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:14,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:17,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:22,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:24,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:24,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:03:24,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:03:28,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:29,523 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:03:33,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:34,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:38,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:39,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:03:39,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:03:43,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:44,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:49,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:54,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:03:54,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:03:54,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:03:55,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:03:59,520 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:04:00,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:04,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:05,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:09,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:04:09,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:04:11,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:14,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:16,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:21,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:24,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:24,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:04:24,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:04:27,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:29,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:04:32,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:34,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:37,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:39,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:04:39,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:04:43,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:44,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:45,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:04:45,477 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:04:45,477 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:04:45,477 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:04:45,846 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:04:46,847 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:04:48,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:53,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:04:54,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:04:54,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:04:54,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:04:59,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:04:59,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:04,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:04,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:09,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:05:09,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:05:10,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:14,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:15,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:20,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:24,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:05:24,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:05:24,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:26,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:29,528 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:05:31,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:34,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:36,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:39,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:05:39,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:05:42,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:44,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:47,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:52,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:54,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:05:54,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:05:54,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:05:58,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:05:59,530 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:06:04,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:04,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:09,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:06:09,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:06:10,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:14,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:15,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:20,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:24,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:06:24,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:06:24,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:26,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:29,533 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:06:31,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:34,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:36,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:39,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:06:39,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:06:42,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:44,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:47,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:06:47,032 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:06:47,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:06:47,033 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:06:47,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:47,896 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:06:48,897 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:06:52,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:54,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:06:54,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:06:54,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:06:57,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:06:59,536 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:07:02,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:04,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:08,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:09,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:07:09,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:07:13,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:14,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:19,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:24,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:07:24,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:07:24,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:25,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:29,539 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:07:30,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:34,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:35,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:39,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:07:39,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:07:41,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:44,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:46,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:52,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:54,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:07:54,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:07:54,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:07:58,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:07:59,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:08:03,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:04,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:08,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:09,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:08:09,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:08:14,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:14,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:20,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:24,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:08:24,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:08:24,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:26,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:29,544 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:08:31,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:34,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:36,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:39,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:08:39,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:08:42,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:44,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:47,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:50,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:08:50,124 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:08:50,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:08:50,126 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:08:50,949 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:08:50,949 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:08:53,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:08:54,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:08:54,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:08:54,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:08:59,547 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:08:59,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:04,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:04,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:09,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:09:09,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:09:10,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:14,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:15,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:21,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:24,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:09:24,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:09:24,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:27,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:29,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:09:32,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:34,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:37,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:39,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:09:39,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:09:42,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:44,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:47,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:52,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:54,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:09:54,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:09:54,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:09:58,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:09:59,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:10:04,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:04,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:09,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:10:09,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:10:10,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:14,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:15,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:20,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:24,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:10:24,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:10:24,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:26,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:29,553 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:10:31,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:34,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:36,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:39,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:10:39,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:10:42,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:44,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:47,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:51,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:10:51,719 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:10:51,720 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:10:51,720 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:10:52,000 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:10:52,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:53,001 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:10:54,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:10:54,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:10:54,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:10:58,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:10:59,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:11:04,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:04,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:09,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:11:09,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:11:10,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:14,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:15,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:20,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:24,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:11:24,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:11:24,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:26,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:29,558 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:11:31,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:34,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:36,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:39,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:11:39,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:11:42,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:44,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:47,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:53,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:54,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:11:54,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:11:54,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:11:58,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:11:59,560 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:12:03,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:04,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:09,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:12:09,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:12:09,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:14,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:14,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:19,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:24,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:12:24,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:12:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:25,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:29,562 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:12:30,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:34,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:35,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:39,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:12:39,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:12:41,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:44,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:46,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:51,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:53,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:12:53,263 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:12:53,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:12:53,264 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:12:54,046 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:12:54,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:12:54,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:12:54,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:12:55,047 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:12:57,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:12:59,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:13:02,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:04,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:07,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:09,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:13:09,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:13:12,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:14,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:17,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:23,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:24,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:13:24,299 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:13:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:28,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:29,566 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:13:34,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:34,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:39,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:13:39,300 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:13:40,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:44,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:45,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:51,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:54,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:13:54,300 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:13:54,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:13:57,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:13:59,570 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:14:02,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:04,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:07,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:09,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:14:09,300 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:14:13,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:14,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:18,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:23,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:24,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:14:24,301 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:14:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:29,573 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:14:29,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:34,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:34,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:39,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:14:39,301 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:14:40,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:44,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:45,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:51,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:54,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:14:54,301 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:14:54,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:14:56,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:14:56,322 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:14:56,323 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:14:56,324 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:14:57,090 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:14:57,090 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:14:57,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:14:59,575 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:15:02,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:04,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:07,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:09,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:15:09,302 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:15:13,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:14,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:18,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:23,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:24,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:15:24,303 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:15:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:29,578 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:15:29,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:34,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:34,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:39,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:15:39,303 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:15:40,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:44,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:45,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:51,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:54,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:15:54,303 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:15:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:15:57,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:15:59,580 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:16:02,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:04,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:07,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:09,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:16:09,303 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:16:12,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:14,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:17,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:22,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:24,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:16:24,303 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:16:24,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:28,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:29,583 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:16:33,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:34,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:39,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:16:39,304 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:16:39,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:44,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:44,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:50,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:54,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:16:54,304 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:16:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:16:56,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:16:57,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:16:57,938 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:16:57,938 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:16:57,939 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:16:58,143 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:16:59,144 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:16:59,584 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:17:01,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:04,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:06,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:09,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:17:09,304 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:17:12,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:14,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:17,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:23,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:24,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:17:24,304 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:17:24,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:29,591 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:17:29,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:34,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:34,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:39,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:17:39,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:17:40,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:44,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:45,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:51,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:54,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:17:54,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:17:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:17:57,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:17:59,589 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:18:02,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:04,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:07,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:09,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:18:09,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:18:12,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:14,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:17,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:22,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:24,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:18:24,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:18:24,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:28,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:29,593 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:18:33,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:34,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:38,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:39,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:18:39,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:18:44,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:44,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:50,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:54,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:18:54,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:18:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:18:56,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:18:59,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:18:59,502 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:18:59,502 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:18:59,503 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:18:59,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:19:00,202 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:19:01,203 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:19:01,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:04,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:06,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:09,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:19:09,305 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:19:12,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:14,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:17,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:23,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:24,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:19:24,306 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:19:24,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:28,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:29,597 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:19:34,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:34,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:39,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:19:39,306 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:19:40,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:44,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:45,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:50,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:54,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:19:54,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:19:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:19:56,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:19:59,600 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:20:01,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:04,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:06,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:09,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:20:09,306 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:20:12,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:14,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:17,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:22,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:24,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:20:24,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:20:24,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:28,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:29,602 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:20:33,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:34,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:38,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:20:39,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:20:43,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:44,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:49,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:54,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:20:54,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:20:54,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:20:55,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:20:59,605 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:21:00,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:02,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:21:02,632 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:21:02,633 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:21:02,633 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:21:03,257 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:21:04,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:05,258 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:21:05,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:09,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:21:09,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:21:11,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:14,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:16,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:21,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:24,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:21:24,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:21:24,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:27,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:29,607 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:21:32,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:34,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:37,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:21:39,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:21:43,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:44,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:48,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:53,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:21:54,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:21:54,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:21:54,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:21:59,613 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:21:59,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:04,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:04,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:09,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:22:09,308 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:22:10,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:14,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:15,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:20,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:24,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:22:24,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:22:24,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:26,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:29,612 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:22:31,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:34,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:36,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:39,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:22:39,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:22:42,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:44,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:47,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:52,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:54,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:22:54,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:22:54,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:22:57,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:22:59,616 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:23:02,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:04,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:23:04,114 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:23:04,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:23:04,115 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:23:04,309 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:23:04,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:05,310 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:23:07,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:09,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:23:09,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:23:13,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:14,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:18,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:24,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:23:24,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:23:24,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:24,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:29,617 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:23:29,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:34,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:34,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:39,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:23:39,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:23:40,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:44,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:45,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:50,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:54,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:23:54,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:23:54,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:23:56,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:23:59,619 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:24:01,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:04,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:06,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:09,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:24:09,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:24:12,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:14,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:17,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:23,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:24,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:24:24,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:24:24,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:28,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:29,621 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:24:33,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:34,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:39,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:24:39,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:24:39,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:44,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:44,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:49,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:54,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:24:54,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:24:54,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:24:55,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:24:59,623 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:25:00,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:04,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:05,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:05,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:25:05,772 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:25:05,772 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:25:05,773 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:25:06,356 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:25:07,357 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:25:09,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:25:09,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:25:11,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:14,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:16,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:22,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:24,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:25:24,311 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:25:24,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:27,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:29,624 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:25:32,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:34,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:37,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:39,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:25:39,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:25:43,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:44,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:48,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:54,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:25:54,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:25:54,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:25:54,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:25:59,627 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:25:59,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:04,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:04,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:09,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:26:09,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:26:10,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:14,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:15,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:20,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:24,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:26:24,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:26:24,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:26,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:29,629 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:26:31,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:34,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:36,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:39,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:26:39,313 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:26:42,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:44,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:47,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:53,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:54,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:26:54,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:26:54,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:26:58,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:26:59,635 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:27:03,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:04,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:08,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:08,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:27:08,803 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:27:08,803 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:27:08,804 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:27:09,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:27:09,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:27:09,401 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:27:11,403 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:27:14,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:14,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:19,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:24,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:27:24,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:27:24,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:25,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:29,633 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:27:30,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:34,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:35,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:39,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:27:39,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:27:41,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:44,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:46,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:51,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:54,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:27:54,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:27:54,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:27:57,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:27:59,637 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:28:02,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:04,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:07,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:09,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:28:09,314 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:28:12,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:14,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:18,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:24,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:28:24,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:28:24,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:24,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:29,639 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:28:29,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:34,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:34,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:39,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:28:39,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:28:40,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:44,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:45,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:51,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:54,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:28:54,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:28:54,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:28:57,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:28:59,641 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:29:02,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:04,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:07,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:09,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:29:09,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:29:10,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:29:10,347 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:29:10,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:29:10,347 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:29:10,454 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:29:11,455 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:29:12,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:14,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:17,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:22,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:24,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:29:24,325 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:29:24,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:27,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:29,643 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:29:32,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:34,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:38,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:39,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:29:39,325 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:29:44,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:44,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:50,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:54,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:29:54,325 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:29:54,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:29:56,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:29:59,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:30:01,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:04,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:06,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:09,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:30:09,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:30:12,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:14,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:17,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:22,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:24,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:30:24,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:30:24,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:28,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:29,647 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:30:33,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:34,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:39,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:30:39,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:30:39,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:44,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:44,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:50,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:54,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:30:54,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:30:54,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:30:56,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:30:59,650 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:31:01,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:04,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:06,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:09,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:31:09,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:31:11,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:31:11,799 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:31:11,799 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:31:11,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:11,800 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:31:12,503 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:31:13,504 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:31:14,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:17,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:23,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:24,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:31:24,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:31:24,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:28,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:29,656 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:31:34,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:34,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:39,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:31:39,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:31:40,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:44,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:45,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:50,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:54,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:31:54,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:31:54,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:31:56,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:31:59,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:32:01,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:04,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:06,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:09,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:32:09,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:32:12,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:14,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:17,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:23,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:24,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:32:24,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:32:24,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:29,659 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:32:29,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:34,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:34,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:39,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:32:39,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:32:40,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:44,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:45,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:51,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:54,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:32:54,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:32:54,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:32:57,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:32:59,660 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:33:02,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:04,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:07,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:09,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:33:09,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:33:12,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:13,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:33:13,301 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:33:13,301 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:33:13,302 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:33:13,550 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:33:14,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:15,551 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:33:18,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:23,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:24,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:33:24,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:33:24,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:29,663 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:33:29,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:34,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:34,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:39,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:33:39,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:33:40,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:44,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:45,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:51,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:54,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:33:54,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:33:54,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:33:57,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:33:59,665 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:34:02,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:04,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:07,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:09,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:34:09,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:34:12,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:14,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:17,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:22,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:34:24,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:34:24,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:28,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:29,667 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:34:33,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:34,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:39,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:34:39,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:34:39,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:44,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:44,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:50,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:54,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:34:54,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:34:54,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:34:56,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:34:59,669 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:35:01,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:04,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:06,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:09,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:35:09,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:35:12,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:14,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:16,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:35:16,337 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:35:16,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:35:16,338 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:35:16,611 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:35:17,611 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:35:17,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:23,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:24,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:35:24,346 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:35:24,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:29,672 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:35:29,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:34,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:34,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:39,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:35:39,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:35:40,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:44,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:45,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:50,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:54,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:35:54,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:35:54,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:35:56,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:35:59,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:36:01,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:04,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:06,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:09,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:36:09,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:36:12,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:14,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:17,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:22,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:24,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:36:24,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:36:24,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:28,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:29,681 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:36:33,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:34,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:39,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:36:39,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:36:39,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:44,426 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:44,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:49,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:54,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:36:54,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:36:54,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:36:55,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:36:59,679 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:37:00,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:04,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:05,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:09,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:37:09,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:37:11,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:14,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:16,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:17,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:37:17,862 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:37:17,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:37:17,864 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:37:18,659 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:37:19,660 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:37:21,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:24,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:37:24,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:37:24,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:27,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:29,681 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:37:32,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:34,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:37,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:39,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:37:39,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:37:43,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:44,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:49,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:54,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:37:54,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:37:54,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:37:55,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:37:59,683 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:38:00,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:04,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:05,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:09,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:38:09,345 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:38:11,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:14,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:16,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:21,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:24,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:38:24,346 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:38:24,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:27,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:29,686 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:38:32,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:34,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:37,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:39,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:38:39,346 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:38:43,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:44,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:48,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:38:54,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:38:54,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:38:54,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:38:59,688 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:38:59,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:04,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:04,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:09,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:39:09,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:39:10,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:14,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:15,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:19,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:39:19,327 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:39:19,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:39:19,328 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:39:19,730 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:39:21,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:21,733 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:39:24,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:39:24,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:39:24,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:26,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:29,690 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:39:31,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:34,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:36,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:39,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:39:39,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:39:42,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:44,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:47,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:53,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:39:54,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:39:54,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:39:54,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:39:59,693 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:39:59,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:04,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:04,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:09,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:40:09,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:40:10,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:14,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:15,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:20,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:24,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:40:24,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:40:24,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:26,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:29,695 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:40:31,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:34,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:36,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:39,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:40:39,349 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:40:42,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:44,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:47,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:53,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:40:54,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:40:54,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:40:54,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:40:59,701 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:40:59,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:04,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:04,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:09,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:41:09,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:41:10,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:14,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:15,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:20,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:22,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:41:22,264 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:41:22,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:41:22,265 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:41:22,818 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:41:23,819 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:41:24,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:41:24,350 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:41:24,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:26,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:29,700 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:41:31,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:34,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:36,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:39,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:41:39,356 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:41:42,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:44,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:47,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:53,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:41:54,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:41:54,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:41:54,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:41:59,704 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:41:59,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:04,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:04,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:09,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:42:09,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:42:10,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:14,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:15,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:21,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:24,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:42:24,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:42:24,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:27,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:29,704 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:42:32,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:34,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:37,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:39,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:42:39,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:42:43,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:44,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:49,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:54,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:42:54,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:42:54,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:42:55,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:42:59,706 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:43:00,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:04,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:05,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:09,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:43:09,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:43:11,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:14,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:16,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:22,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:23,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:43:23,665 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:43:23,666 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:43:23,666 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:43:23,886 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:43:24,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:43:24,357 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:43:24,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:25,888 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:43:27,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:29,708 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:43:33,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:34,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:38,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:39,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:43:39,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:43:44,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:44,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:50,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:54,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:43:54,358 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:43:54,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:43:56,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:43:59,711 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:44:01,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:04,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:06,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:09,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:44:09,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:44:12,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:14,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:17,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:23,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:24,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:44:24,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:44:24,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:28,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:29,713 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:44:34,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:34,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:39,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:44:39,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:44:40,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:44,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:45,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:50,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:54,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:44:54,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:44:54,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:44:56,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:44:59,715 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:45:01,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:04,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:06,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:09,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:45:09,359 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:45:12,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:14,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:17,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:22,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:24,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:45:24,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:45:24,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:25,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:45:25,444 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:45:25,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:45:25,446 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:45:25,942 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:45:27,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:27,944 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:45:29,722 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:45:33,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:34,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:39,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:45:39,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:45:39,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:44,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:44,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:50,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:54,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:45:54,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:45:54,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:45:56,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:45:59,720 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:46:01,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:04,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:06,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:09,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:46:09,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:46:12,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:14,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:17,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:22,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:24,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:46:24,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:46:24,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:27,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:29,724 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:46:32,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:34,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:38,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:39,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:46:39,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:46:44,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:44,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:49,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:54,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:46:54,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:46:54,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:46:55,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:46:59,725 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:47:00,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:04,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:05,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:09,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:47:09,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:47:11,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:14,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:16,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:21,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:24,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:47:24,361 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:47:24,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:27,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:28,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:47:28,734 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:47:28,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:47:28,735 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:47:29,008 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:47:29,727 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:47:30,009 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:47:32,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:34,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:37,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:39,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:47:39,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:47:43,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:44,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:49,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:54,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:47:54,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:47:54,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:47:55,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:47:59,730 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:48:00,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:04,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:05,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:09,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:48:09,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:48:11,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:14,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:16,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:22,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:24,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:48:24,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:48:24,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:28,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:29,732 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:48:34,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:34,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:39,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:48:39,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:48:40,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:44,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:45,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:50,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:54,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:48:54,362 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:48:54,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:48:56,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:48:59,734 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:49:01,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:04,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:06,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:09,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:49:09,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:49:12,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:14,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:17,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:23,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:24,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:49:24,363 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:49:24,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:28,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:29,736 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:49:30,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:49:30,241 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:49:30,241 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:49:30,242 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:49:31,056 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:49:32,057 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:49:34,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:34,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:39,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:49:39,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:49:40,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:44,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:45,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:50,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:54,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:49:54,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:49:54,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:49:56,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:49:59,739 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:50:01,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:04,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:06,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:09,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:50:09,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:50:12,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:14,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:17,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:23,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:24,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:50:24,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:50:24,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:29,745 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:50:29,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:34,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:34,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:39,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:50:39,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:50:40,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:44,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:45,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:51,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:54,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:50:54,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:50:54,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:50:57,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:50:59,743 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:51:02,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:04,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:07,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:09,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:51:09,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:51:13,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:14,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:19,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:24,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:51:24,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:51:24,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:25,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:29,746 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:51:30,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:31,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:51:31,751 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:51:31,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:51:31,752 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:51:32,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:51:34,103 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:51:34,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:35,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:39,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:51:39,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:51:41,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:44,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:46,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:51,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:54,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:51:54,366 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:51:54,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:51:57,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:51:59,748 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:52:02,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:04,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:07,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:09,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:52:09,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:52:13,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:14,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:18,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:23,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:24,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:52:24,367 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:52:24,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:29,750 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:52:29,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:34,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:34,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:39,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:52:39,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:52:40,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:44,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:45,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:51,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:54,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:52:54,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:52:54,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:52:57,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:52:59,752 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:53:02,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:04,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:08,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:09,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:53:09,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:53:14,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:14,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:20,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:24,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:53:24,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:53:24,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:26,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:29,755 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:53:31,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:34,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:34,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:53:34,788 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:53:34,789 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:53:34,789 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:53:35,152 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:53:36,153 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:53:36,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:39,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:53:39,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:53:42,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:44,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:47,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:52,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:54,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:53:54,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:53:54,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:53:58,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:53:59,757 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:54:03,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:04,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:08,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:09,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:54:09,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:54:13,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:14,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:19,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:24,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:54:24,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:54:24,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:25,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:29,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:54:30,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:34,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:36,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:39,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:54:39,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:54:42,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:44,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:47,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:53,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:54,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:54:54,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:54:54,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:54:59,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:54:59,766 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:55:04,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:04,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:09,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:55:09,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:55:10,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:14,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:15,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:21,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:24,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:55:24,369 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:55:24,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:27,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:29,764 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:55:32,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:34,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:36,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:55:36,388 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:55:36,389 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:55:36,391 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:55:37,208 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:55:38,209 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:55:38,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:39,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:55:39,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:55:43,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:44,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:48,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:54,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:55:54,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:55:54,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:55:54,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:59,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:55:59,766 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:56:04,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:04,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:09,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:56:09,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:56:10,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:14,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:15,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:21,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:24,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:56:24,370 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:56:24,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:27,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:29,770 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:56:32,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:34,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:38,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:39,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:56:39,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:56:44,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:44,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:49,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:54,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:56:54,371 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:56:54,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:56:55,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:56:59,771 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:57:00,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:04,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:06,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:09,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:57:09,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:57:12,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:14,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:17,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:23,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:24,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:57:24,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:57:24,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:29,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:29,773 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:57:34,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:34,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:37,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:57:37,998 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:57:37,998 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:57:37,999 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:57:38,259 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:57:39,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:57:39,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:57:40,260 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:57:40,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:44,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:45,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:50,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:54,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:57:54,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:57:54,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:57:56,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:57:59,775 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:58:01,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:04,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:07,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:09,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:58:09,372 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:58:13,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:14,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:18,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:24,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:58:24,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:58:24,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:24,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:29,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:29,777 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:58:34,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:35,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:39,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:58:39,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:58:41,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:44,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:46,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:52,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:54,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:58:54,373 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:58:54,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:58:58,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:58:59,780 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:59:03,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:04,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:08,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:09,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:59:09,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:59:13,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:14,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:19,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:24,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:59:24,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:59:24,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:25,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:29,787 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 11:59:30,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:34,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:35,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:39,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:59:39,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:59:41,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 11:59:41,043 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 11:59:41,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 11:59:41,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:41,045 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 11:59:41,312 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 11:59:41,313 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 11:59:44,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:46,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:52,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:54,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 11:59:54,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 11:59:54,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 11:59:58,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 11:59:59,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:00:03,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:04,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:08,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:09,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:00:09,374 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:00:13,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:14,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:19,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:24,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:00:24,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:00:24,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:25,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:29,789 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:00:30,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:34,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:35,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:39,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:00:39,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:00:41,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:44,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:46,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:51,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:54,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:00:54,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:00:54,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:00:57,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:00:59,789 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:01:02,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:04,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:07,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:09,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:01:09,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:01:13,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:14,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:18,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:24,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:01:24,375 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:01:24,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:24,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:29,792 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:01:29,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:34,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:34,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:39,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:01:39,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:01:40,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:42,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:01:42,734 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:01:42,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:01:42,735 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:01:43,383 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:01:43,383 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:01:44,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:45,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:50,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:54,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:01:54,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:01:54,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:01:56,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:01:59,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:02:01,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:04,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:06,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:09,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:02:09,376 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:02:12,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:14,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:17,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:23,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:24,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:02:24,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:02:24,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:28,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:29,797 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:02:34,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:34,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:39,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:02:39,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:02:40,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:44,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:45,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:50,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:54,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:02:54,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:02:54,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:02:56,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:02:59,799 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:03:01,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:04,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:06,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:09,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:03:09,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:03:12,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:14,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:17,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:23,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:24,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:03:24,377 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:03:24,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:29,801 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:03:29,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:34,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:34,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:39,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:03:39,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:03:40,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:44,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:03:44,247 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:03:44,248 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:03:44,249 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:03:44,435 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:03:44,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:45,436 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:03:46,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:51,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:54,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:03:54,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:03:54,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:03:57,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:03:59,808 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:04:02,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:04,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:07,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:09,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:04:09,378 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:04:13,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:14,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:18,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:24,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:04:24,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:04:24,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:24,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:29,806 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:04:29,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:34,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:34,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:39,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:04:39,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:04:40,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:44,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:45,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:51,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:54,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:04:54,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:04:54,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:04:57,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:04:59,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:05:02,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:04,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:07,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:09,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:05:09,379 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:05:13,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:14,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:18,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:23,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:24,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:05:24,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:05:24,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:29,811 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:05:29,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:34,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:34,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:39,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:05:39,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:05:40,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:44,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:45,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:47,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:05:47,317 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:05:47,318 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:05:47,318 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:05:47,494 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:05:49,496 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:05:51,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:54,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:05:54,382 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:05:54,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:05:56,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:05:59,814 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:06:01,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:04,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:07,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:09,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:06:09,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:06:13,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:14,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:18,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:23,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:24,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:06:24,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:06:24,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:28,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:29,816 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:06:34,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:34,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:39,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:06:39,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:06:40,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:44,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:45,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:50,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:54,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:06:54,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:06:54,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:06:56,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:06:59,819 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:07:01,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:04,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:07,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:09,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:07:09,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:07:12,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:14,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:18,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:24,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:07:24,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:07:24,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:24,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:29,821 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:07:29,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:34,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:34,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:39,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:07:39,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:07:39,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:44,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:44,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:48,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:07:48,889 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:07:48,889 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:07:48,890 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:07:49,544 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:07:49,545 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:07:50,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:54,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:07:54,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:07:54,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:07:56,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:07:59,823 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:08:01,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:04,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:07,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:09,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:08:09,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:08:12,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:14,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:18,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:23,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:24,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:08:24,384 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:08:24,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:29,826 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:08:29,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:34,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:34,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:39,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:08:39,385 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:08:40,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:44,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:45,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:51,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:54,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:08:54,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:08:54,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:08:56,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:08:59,833 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:09:02,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:04,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:07,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:09,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:09:09,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:09:13,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:14,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:19,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:24,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:09:24,386 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:09:25,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:25,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:29,831 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:09:30,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:34,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:36,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:39,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:09:39,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:09:41,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:44,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:47,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:50,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:09:50,490 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:09:50,491 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:09:50,493 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:09:50,592 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:09:51,593 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:09:52,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:54,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:09:54,387 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:09:54,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:09:58,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:09:59,833 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:10:03,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:04,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:09,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:09,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:10:09,396 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:10:14,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:15,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:20,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:24,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:10:24,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:10:24,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:26,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:29,837 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:10:32,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:34,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:37,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:39,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:10:39,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:10:43,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:44,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:49,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:54,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:10:54,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:10:54,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:10:54,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:10:59,838 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:10:59,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:04,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:05,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:09,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:11:09,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:11:11,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:14,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:16,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:21,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:24,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:11:24,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:11:24,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:27,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:29,841 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:11:32,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:34,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:38,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:39,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:11:39,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:11:43,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:44,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:49,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:52,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:11:52,258 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:11:52,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:11:52,259 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:11:52,648 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:11:53,649 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:11:54,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:11:54,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:11:54,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:11:54,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:11:59,843 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:11:59,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:04,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:05,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:09,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:12:09,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:12:11,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:14,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:17,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:22,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:24,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:12:24,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:12:24,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:28,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:29,845 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:12:33,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:34,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:39,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:39,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:12:39,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:12:44,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:44,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:49,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:54,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:12:54,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:12:54,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:12:54,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:12:59,848 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:12:59,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:04,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:05,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:09,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:13:09,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:13:10,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:14,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:16,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:21,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:24,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:13:24,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:13:24,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:27,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:29,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:13:32,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:34,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:38,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:39,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:13:39,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:13:43,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:44,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:49,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:54,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:54,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:13:54,399 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:13:54,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:13:55,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:13:55,227 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:13:55,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:13:55,228 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:13:55,700 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:13:55,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:13:59,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:13:59,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:14:04,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:05,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:09,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:14:09,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:14:10,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:14,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:16,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:22,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:24,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:14:24,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:14:24,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:28,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:29,857 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:14:33,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:34,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:39,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:39,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:14:39,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:14:44,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:44,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:49,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:54,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:14:54,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:14:54,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:14:54,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:14:59,858 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:14:59,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:04,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:05,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:09,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:15:09,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:15:11,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:14,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:17,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:22,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:24,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:15:24,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:15:24,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:28,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:29,860 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:15:33,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:34,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:39,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:39,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:15:39,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:15:44,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:44,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:50,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:54,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:15:54,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:15:54,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:15:55,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:15:56,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:15:56,823 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:15:56,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:15:56,824 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:15:57,747 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:15:57,747 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:15:59,862 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:16:00,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:04,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:06,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:09,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:16:09,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:16:12,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:14,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:17,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:22,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:24,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:16:24,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:16:24,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:28,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:29,864 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:16:34,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:34,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:39,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:16:39,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:16:39,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:44,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:45,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:50,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:54,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:16:54,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:16:54,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:16:56,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:16:59,867 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:17:02,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:05,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:07,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:09,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:17:09,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:17:13,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:15,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:19,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:24,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:17:24,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:17:24,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:25,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:29,870 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:17:29,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:35,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:35,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:39,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:17:39,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:17:40,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:45,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:46,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:51,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:54,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:17:54,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:17:55,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:17:57,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:17:58,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:17:58,343 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:17:58,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:17:58,344 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:17:58,806 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:17:59,807 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:17:59,878 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:18:02,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:05,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:08,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:09,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:18:09,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:18:13,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:15,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:19,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:24,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:24,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:18:24,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:18:25,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:29,875 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:18:29,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:35,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:35,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:39,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:18:39,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:18:40,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:45,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:46,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:51,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:54,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:18:54,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:18:55,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:18:57,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:18:59,880 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:19:02,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:05,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:07,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:09,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:19:09,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:19:13,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:15,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:19,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:24,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:24,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:19:24,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:19:25,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:29,881 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:19:29,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:35,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:35,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:39,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:19:39,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:19:40,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:45,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:46,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:51,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:54,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:19:54,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:19:55,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:19:57,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:19:59,883 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:20:01,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:20:01,479 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:20:01,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:20:01,480 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:20:01,873 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:20:01,873 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:20:02,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:05,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:08,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:09,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:20:09,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:20:13,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:15,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:19,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:24,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:24,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:20:24,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:20:25,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:29,885 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:20:29,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:35,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:35,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:39,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:20:39,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:20:40,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:45,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:46,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:51,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:54,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:20:54,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:20:55,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:20:57,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:20:59,888 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:21:02,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:05,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:08,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:09,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:21:09,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:21:14,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:15,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:20,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:24,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:21:24,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:21:25,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:26,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:29,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:21:31,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:35,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:36,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:39,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:21:39,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:21:41,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:45,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:47,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:52,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:54,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:21:54,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:21:55,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:21:57,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:21:59,892 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:22:03,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:22:03,047 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:22:03,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:22:03,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:03,049 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:22:03,925 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:22:03,926 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:22:05,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:08,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:09,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:22:09,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:22:13,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:15,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:19,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:24,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:24,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:22:24,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:22:25,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:29,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:22:29,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:35,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:35,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:39,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:22:39,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:22:40,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:45,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:45,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:50,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:54,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:22:54,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:22:55,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:22:56,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:22:59,901 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:23:01,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:05,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:07,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:09,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:23:09,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:23:12,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:15,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:18,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:24,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:24,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:23:24,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:23:25,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:29,899 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:23:29,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:35,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:35,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:39,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:23:39,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:23:40,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:45,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:46,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:51,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:54,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:23:54,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:23:55,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:23:56,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:23:59,904 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:24:02,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:04,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:24:04,756 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:24:04,757 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:24:04,758 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:24:04,981 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:24:05,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:05,982 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:24:08,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:09,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:24:09,427 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:24:13,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:15,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:19,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:24,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:24,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:24:24,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:24:25,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:29,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:24:29,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:35,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:35,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:39,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:24:39,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:24:40,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:45,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:46,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:51,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:54,427 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:24:54,428 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:24:55,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:24:57,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:24:59,908 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:25:02,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:05,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:07,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:09,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:25:09,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:25:13,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:15,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:19,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:24,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:25:24,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:25:24,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:25,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:29,911 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:25:29,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:35,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:35,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:39,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:25:39,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:25:41,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:45,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:46,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:51,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:54,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:25:54,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:25:55,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:25:57,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:25:59,913 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:26:02,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:05,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:07,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:26:07,953 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:26:07,954 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:26:07,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:07,955 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:26:08,038 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:26:09,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:26:09,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:26:10,040 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:26:13,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:15,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:19,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:24,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:26:24,429 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:26:24,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:25,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:29,916 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:26:29,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:35,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:35,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:39,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:26:39,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:26:40,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:45,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:46,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:52,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:54,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:26:54,430 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:26:55,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:26:57,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:26:59,918 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:27:02,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:05,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:08,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:09,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:27:09,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:27:14,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:15,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:19,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:24,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:27:24,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:27:24,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:25,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:29,924 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:27:29,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:35,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:35,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:39,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:27:39,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:27:41,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:45,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:47,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:53,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:54,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:27:54,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:27:55,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:27:59,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:27:59,923 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:28:04,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:05,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:09,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:28:09,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:28:09,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:28:09,614 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:28:09,615 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:28:09,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:09,618 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:28:10,096 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:28:10,096 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:28:15,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:15,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:21,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:24,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:28:24,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:28:25,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:27,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:29,925 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:28:32,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:35,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:38,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:39,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:28:39,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:28:44,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:45,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:49,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:54,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:28:54,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:28:54,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:28:55,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:28:59,929 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:28:59,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:05,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:05,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:09,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:29:09,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:29:10,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:15,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:16,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:22,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:24,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:29:24,432 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:29:25,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:27,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:29,929 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:29:33,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:35,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:39,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:29:39,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:29:39,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:45,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:45,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:51,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:54,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:29:54,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:29:55,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:29:56,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:29:59,931 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:30:02,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:05,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:07,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:09,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:30:09,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:30:11,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:30:11,088 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:30:11,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:30:11,089 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:30:11,155 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:30:12,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:30:13,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:15,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:18,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:24,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:30:24,433 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:30:24,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:25,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:29,934 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:30:29,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:35,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:35,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:39,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:30:39,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:30:40,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:45,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:46,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:51,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:54,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:30:54,435 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:30:55,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:30:57,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:30:59,936 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:31:02,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:05,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:08,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:09,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:31:09,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:31:13,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:15,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:19,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:24,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:31:24,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:31:24,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:25,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:29,939 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:31:29,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:35,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:35,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:39,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:31:39,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:31:41,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:45,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:46,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:51,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:54,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:31:54,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:31:55,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:31:57,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:31:59,945 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:32:02,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:05,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:08,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:09,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:32:09,452 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:32:13,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:13,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:32:13,979 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:32:13,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:32:13,980 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:32:14,220 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:32:15,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:16,222 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:32:19,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:24,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:32:24,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:32:24,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:25,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:29,943 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:32:29,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:35,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:35,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:39,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:32:39,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:32:40,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:45,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:46,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:51,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:54,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:32:54,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:32:55,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:32:57,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:32:59,947 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:33:02,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:05,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:07,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:09,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:33:09,453 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:33:12,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:15,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:18,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:23,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:24,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:33:24,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:33:25,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:29,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:29,948 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:33:34,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:35,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:39,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:33:39,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:33:39,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:45,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:45,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:50,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:54,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:33:54,454 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:33:55,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:33:55,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:33:59,950 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:34:00,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:05,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:05,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:09,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:34:09,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:34:11,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:15,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:15,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:34:15,799 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:34:15,800 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:34:15,800 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:34:16,279 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:34:16,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:18,281 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:34:22,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:24,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:34:24,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:34:25,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:28,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:29,953 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:34:33,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:35,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:38,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:39,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:34:39,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:34:44,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:45,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:50,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:54,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:34:54,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:34:55,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:34:55,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:34:59,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:35:00,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:05,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:05,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:09,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:35:09,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:35:11,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:15,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:16,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:22,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:24,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:35:24,455 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:35:25,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:27,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:29,957 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:35:33,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:35,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:39,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:35:39,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:35:39,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:45,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:45,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:51,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:54,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:35:54,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:35:55,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:35:57,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:35:59,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:36:02,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:05,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:08,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:09,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:36:09,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:36:13,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:15,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:17,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:36:17,510 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:36:17,510 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:36:17,511 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:36:18,334 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:36:18,334 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:36:19,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:24,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:36:24,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:36:24,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:25,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:29,966 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:36:29,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:35,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:35,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:39,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:36:39,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:36:41,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:45,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:47,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:52,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:54,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:36:54,456 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:36:55,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:36:58,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:36:59,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:37:04,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:05,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:09,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:37:09,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:37:09,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:15,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:15,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:20,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:24,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:37:24,457 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:37:25,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:25,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:29,970 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:37:30,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:35,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:35,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:39,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:37:39,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:37:41,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:45,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:47,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:53,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:54,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:37:54,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:37:55,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:37:58,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:37:59,970 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:38:04,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:05,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:09,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:38:09,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:38:09,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:15,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:15,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:20,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:38:20,826 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:38:20,826 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:38:20,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:20,828 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:38:21,442 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:38:22,443 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:38:24,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:38:24,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:38:25,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:26,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:29,973 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:38:32,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:35,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:37,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:39,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:38:39,458 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:38:43,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:45,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:48,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:54,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:38:54,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:38:54,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:38:55,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:38:59,975 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:38:59,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:05,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:05,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:09,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:39:09,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:39:10,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:15,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:15,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:20,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:24,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:39:24,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:39:25,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:26,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:29,977 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:39:31,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:35,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:36,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:39,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:39:39,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:39:42,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:45,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:48,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:53,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:54,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:39:54,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:39:55,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:39:58,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:39:59,980 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:40:04,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:05,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:09,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:40:09,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:40:09,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:15,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:15,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:20,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:22,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:40:22,307 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:40:22,307 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:40:22,308 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:40:22,527 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:40:24,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:40:24,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:40:24,529 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:40:25,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:26,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:29,982 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:40:31,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:35,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:37,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:39,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:40:39,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:40:42,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:45,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:48,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:53,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:54,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:40:54,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:40:55,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:40:59,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:40:59,984 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:41:04,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:05,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:09,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:41:09,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:41:09,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:15,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:15,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:20,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:24,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:41:24,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:41:25,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:25,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:29,990 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:41:30,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:35,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:36,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:39,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:41:39,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:41:41,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:45,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:46,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:52,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:54,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:41:54,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:41:55,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:41:58,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:41:59,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:42:03,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:05,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:08,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:09,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:42:09,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:42:14,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:15,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:19,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:23,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:42:23,740 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:42:23,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:42:23,742 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:42:24,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:42:24,461 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:42:24,592 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:42:24,592 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:42:25,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:25,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:29,991 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:42:30,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:35,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:35,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:39,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:42:39,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:42:41,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:45,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:46,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:51,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:54,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:42:54,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:42:55,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:42:57,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:42:59,995 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:43:02,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:05,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:08,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:09,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:43:09,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:43:13,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:15,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:18,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:23,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:24,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:43:24,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:43:25,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:29,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:29,996 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:43:35,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:35,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:39,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:43:39,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:43:40,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:45,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:46,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:51,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:54,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:43:54,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:43:55,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:43:57,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:43:59,999 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:44:02,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:05,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:08,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:09,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:44:09,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:44:14,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:15,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:19,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:24,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:44:24,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:44:24,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:25,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:44:25,328 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:44:25,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:44:25,329 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:44:25,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:25,645 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:44:26,645 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:44:30,001 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:44:30,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:35,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:35,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:39,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:44:39,464 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:44:41,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:45,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:46,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:52,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:44:54,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:44:54,465 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:44:55,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:44:58,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:00,004 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:45:03,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:05,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:08,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:09,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:45:09,479 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:45:14,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:15,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:19,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:24,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:45:24,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:45:25,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:25,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:30,006 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:45:30,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:35,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:35,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:39,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:45:39,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:45:41,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:45,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:47,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:52,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:45:54,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:45:54,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:45:55,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:45:57,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:00,012 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:46:03,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:05,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:46:08,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:09,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:46:09,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:46:14,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:15,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:46:20,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:24,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:46:24,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:46:25,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:46:26,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:28,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:46:28,288 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:46:28,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:46:28,289 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:46:28,689 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:46:30,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:46:30,690 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:46:31,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:35,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:46:37,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:39,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:46:39,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:46:42,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:45,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:46:48,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:54,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:46:54,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:46:54,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:46:55,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:00,013 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:47:00,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:05,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:05,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:09,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:47:09,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:47:11,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:15,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:16,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:21,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:24,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:47:24,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:47:25,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:26,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:30,018 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:47:32,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:35,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:37,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:39,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:47:39,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:47:43,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:45,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:47:49,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:54,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:47:54,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:47:54,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:47:55,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:00,018 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:48:00,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:05,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:05,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:09,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:48:09,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:48:10,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:15,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:15,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:21,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:24,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:48:24,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:48:25,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:27,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:30,021 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:48:30,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:48:30,051 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:48:30,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:48:30,052 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:48:30,743 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:48:32,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:32,744 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:48:35,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:38,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:39,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:48:39,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:48:44,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:45,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:50,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:48:54,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:48:54,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:48:55,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:48:56,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:00,023 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:49:01,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:05,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:07,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:09,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:49:09,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:49:12,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:15,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:18,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:24,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:49:24,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:49:24,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:25,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:30,025 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:49:30,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:35,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:35,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:39,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:49:39,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:49:40,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:45,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:46,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:52,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:49:54,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:49:54,484 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:49:55,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:49:57,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:00,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:50:02,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:05,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:08,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:09,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:50:09,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:50:14,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:15,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:20,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:24,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:50:24,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:50:25,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:26,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:30,033 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:50:31,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:31,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:50:31,683 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:50:31,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:50:31,684 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:50:31,791 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:50:32,792 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:50:35,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:37,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:39,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:50:39,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:50:42,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:45,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:48,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:53,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:50:54,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:50:54,485 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:50:55,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:50:59,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:00,032 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:51:04,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:05,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:09,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:51:09,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:51:09,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:15,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:15,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:20,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:24,485 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:51:24,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:51:25,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:26,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:30,036 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:51:31,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:35,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:37,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:39,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:51:39,486 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:51:42,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:45,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:48,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:53,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:51:54,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:51:54,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:51:55,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:51:59,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:00,036 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:52:04,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:05,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:52:09,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:52:09,487 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:52:09,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:15,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:52:15,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:20,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:24,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:52:24,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:52:25,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:52:26,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:30,039 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:52:31,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:34,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:52:34,814 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:52:34,814 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:52:34,815 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:52:34,839 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:52:35,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:52:36,840 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:52:37,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:39,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:52:39,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:52:43,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:45,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:52:49,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:54,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:52:54,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:52:54,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:52:55,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:00,041 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:53:00,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:05,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:05,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:09,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:53:09,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:53:10,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:15,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:16,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:21,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:24,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:53:24,496 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:53:25,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:27,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:30,044 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:53:32,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:35,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:37,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:39,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:53:39,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:53:42,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:45,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:48,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:53,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:53:54,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:53:54,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:53:55,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:53:59,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:00,046 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:54:04,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:05,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:09,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:54:09,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:54:09,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:15,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:15,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:20,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:24,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:54:24,497 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:54:25,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:26,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:30,048 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:54:31,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:35,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:36,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:54:36,420 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:54:36,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:54:36,421 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:54:36,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:54:37,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:38,901 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:54:39,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:54:39,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:54:42,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:45,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:48,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:53,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:54:54,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:54:54,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:54:55,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:54:59,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:00,051 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:55:04,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:05,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:09,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:55:09,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:55:10,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:15,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:15,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:20,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:24,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:55:24,498 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:55:25,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:26,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:30,058 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:55:31,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:35,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:37,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:39,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:55:39,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:55:42,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:45,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:48,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:53,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:55:54,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:55:54,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:55:55,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:55:58,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:00,055 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:56:03,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:05,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:56:09,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:56:09,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:56:09,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:15,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:56:15,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:20,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:24,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:56:24,499 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:56:25,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:56:26,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:30,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:56:31,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:35,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:56:37,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:38,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:56:38,565 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:56:38,565 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:56:38,566 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:56:38,951 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:56:39,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:56:39,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:56:40,952 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:56:43,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:45,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:56:49,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:54,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:56:54,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:56:55,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:56:55,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:00,060 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:57:00,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:05,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:06,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:09,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:57:09,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:57:12,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:15,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:17,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:23,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:24,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:57:24,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:57:25,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:29,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:30,062 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:57:34,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:35,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:39,500 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:57:39,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:57:40,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:45,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:45,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:50,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:57:54,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:57:54,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:57:55,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:57:56,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:00,065 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:58:01,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:05,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:07,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:09,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:58:09,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:58:12,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:15,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:17,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:22,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:24,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:58:24,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:58:25,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:28,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:30,067 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:58:33,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:35,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:39,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:58:39,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:58:39,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:41,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 12:58:41,788 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 12:58:41,788 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 12:58:41,789 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 12:58:42,002 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 12:58:43,002 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 12:58:44,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:45,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:50,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:58:54,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:58:54,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:58:55,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:58:56,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:00,070 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:59:02,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:05,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:08,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:09,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:59:09,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:59:13,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:15,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:19,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:24,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:59:24,504 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:59:24,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:25,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:30,072 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 12:59:30,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:35,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:35,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:39,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:59:39,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:59:40,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:45,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:45,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:50,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 12:59:54,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 12:59:54,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 12:59:55,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 12:59:56,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:00,079 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:00:01,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:05,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:07,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:09,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:00:09,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:00:13,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:15,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:19,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:24,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:00:24,505 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:00:25,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:25,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:30,077 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:00:30,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:35,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:36,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:39,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:00:39,506 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:00:42,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:43,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:00:43,269 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:00:43,270 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:00:43,272 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:00:44,047 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:00:45,048 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:00:45,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:48,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:53,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:00:54,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:00:54,506 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:00:55,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:00:59,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:00,141 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:01:04,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:05,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:01:09,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:01:09,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:01:10,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:15,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:01:15,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:20,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:24,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:01:24,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:01:25,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:01:26,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:30,145 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:01:32,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:35,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:01:38,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:39,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:01:39,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:01:43,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:45,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:01:49,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:54,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:01:54,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:01:55,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:01:55,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:00,146 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:02:00,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:05,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:05,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:09,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:02:09,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:02:11,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:15,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:16,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:21,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:24,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:02:24,507 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:02:25,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:27,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:30,148 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:02:32,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:35,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:38,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:39,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:02:39,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:02:44,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:45,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:02:45,016 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:02:45,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:02:45,017 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:02:45,102 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:02:45,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:47,104 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:02:49,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:02:54,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:02:54,510 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:02:55,109 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:02:55,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:02:55,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:00,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:03:01,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:03,116 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:03:05,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:06,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:09,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:03:09,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:03:11,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:15,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:17,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:22,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:24,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:03:24,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:03:25,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:27,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:30,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:03:33,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:35,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:39,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:03:39,521 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:03:39,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:45,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:45,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:51,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:03:54,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:03:54,521 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:03:55,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:03:56,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:00,155 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:04:02,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:05,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:07,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:09,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:04:09,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:04:12,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:15,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:17,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:22,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:24,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:04:24,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:04:25,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:27,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:30,161 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:04:32,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:35,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:38,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:39,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:04:39,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:04:44,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:45,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:50,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:04:54,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:04:54,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:04:55,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:04:55,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:00,159 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:05:01,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:05,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:05:05,111 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:05:05,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:05:05,113 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:05:05,174 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:05:05,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:07,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:07,175 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:05:09,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:05:09,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:05:12,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:15,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:18,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:23,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:24,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:05:24,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:05:25,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:28,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:30,162 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:05:34,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:35,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:39,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:05:39,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:05:39,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:45,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:45,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:51,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:05:54,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:05:54,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:05:55,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:05:56,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:00,167 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:06:02,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:05,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:07,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:09,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:06:09,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:06:13,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:15,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:18,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:24,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:06:24,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:06:24,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:25,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:30,166 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:06:30,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:35,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:35,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:39,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:06:39,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:06:40,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:45,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:46,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:51,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:06:54,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:06:54,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:06:55,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:06:56,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:00,169 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:07:02,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:05,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:06,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:07:06,728 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:07:06,728 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:07:06,729 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:07:07,219 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:07:07,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:09,220 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:07:09,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:07:09,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:07:12,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:15,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:17,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:23,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:24,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:07:24,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:07:25,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:29,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:30,172 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:07:34,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:35,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:39,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:07:39,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:07:39,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:45,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:45,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:51,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:07:54,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:07:54,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:07:55,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:07:56,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:00,174 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:08:02,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:05,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:07,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:09,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:08:09,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:08:12,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:15,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:17,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:23,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:24,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:08:24,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:08:25,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:28,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:30,176 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:08:34,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:35,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:39,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:08:39,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:08:39,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:45,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:45,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:50,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:08:54,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:08:54,538 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:08:55,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:08:55,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:00,183 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:09:01,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:05,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:06,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:08,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:09:08,397 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:09:08,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:09:08,399 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:09:09,279 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:09:09,279 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:09:09,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:09:09,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:09:11,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:15,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:17,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:22,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:24,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:09:24,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:09:25,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:28,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:30,181 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:09:34,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:35,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:39,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:09:39,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:09:39,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:45,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:45,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:51,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:09:54,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:09:54,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:09:55,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:09:56,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:00,185 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:10:02,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:05,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:07,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:09,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:10:09,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:10:13,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:15,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:18,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:23,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:24,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:10:24,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:10:25,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:29,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:30,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:10:34,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:35,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:39,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:41,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:10:41,855 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:10:45,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:45,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:50,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:55,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:10:56,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:10:56,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:10:56,855 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:11:00,189 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:11:02,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:05,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:07,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:11,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:11:11,856 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:11:12,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:11:12,959 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:11:12,959 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:11:12,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:12,960 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:11:13,328 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:11:15,329 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:11:15,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:18,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:24,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:25,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:26,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:11:26,855 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:11:29,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:30,191 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:11:34,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:35,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:40,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:41,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:11:41,856 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:11:45,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:46,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:52,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:11:55,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:11:56,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:11:56,856 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:11:58,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:00,195 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:12:03,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:05,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:09,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:11,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:12:11,856 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:12:15,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:15,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:20,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:25,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:25,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:26,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:12:26,857 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:12:30,197 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:12:31,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:35,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:36,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:41,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:41,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:12:41,857 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:12:45,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:47,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:52,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:12:55,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:12:56,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:12:56,857 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:12:58,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:00,200 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:13:03,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:05,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:09,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:11,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:13:11,858 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:13:14,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:13:14,866 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:13:14,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:13:14,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:14,867 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:13:15,369 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:13:15,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:17,370 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:13:20,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:25,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:25,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:26,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:13:26,857 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:13:30,206 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:13:31,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:35,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:36,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:41,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:41,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:13:41,857 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:13:45,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:47,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:53,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:13:55,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:13:56,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:13:56,858 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:13:59,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:00,205 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:14:05,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:05,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:11,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:11,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:14:11,858 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:14:15,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:16,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:21,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:25,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:26,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:14:26,858 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:14:27,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:30,210 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:14:32,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:35,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:37,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:41,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:14:41,859 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:14:43,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:45,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:49,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:54,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:14:55,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:14:56,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:14:56,859 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:14:59,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:00,211 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:15:04,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:05,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:10,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:11,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:15:11,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:15:15,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:15,715 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:16,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:15:16,525 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:15:16,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:15:16,526 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:15:17,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:15:19,416 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:15:20,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:25,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:25,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:26,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:15:26,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:15:30,214 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:15:31,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:35,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:36,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:41,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:15:41,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:15:42,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:45,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:47,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:52,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:15:55,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:15:56,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:15:56,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:15:58,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:00,217 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:16:03,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:05,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:09,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:11,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:16:11,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:16:15,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:15,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:21,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:25,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:26,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:16:26,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:16:27,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:30,219 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:16:32,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:35,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:37,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:41,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:16:41,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:16:42,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:45,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:47,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:53,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:16:55,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:16:56,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:16:56,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:16:58,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:00,222 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:17:03,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:05,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:08,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:11,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:17:11,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:17:14,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:15,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:19,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:19,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:17:19,570 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:17:19,570 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:17:19,571 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:17:20,469 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:17:21,470 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:17:24,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:25,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:26,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:17:26,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:17:29,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:30,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:17:34,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:35,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:39,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:41,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:17:41,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:17:44,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:45,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:49,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:54,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:17:55,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:17:56,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:17:56,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:18:00,230 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:18:00,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:05,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:05,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:11,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:11,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:18:11,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:18:15,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:17,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:22,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:25,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:26,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:18:26,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:18:27,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:30,229 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:18:33,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:35,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:38,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:41,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:18:41,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:18:44,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:45,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:49,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:54,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:18:55,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:18:56,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:18:56,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:19:00,233 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:19:00,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:05,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:05,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:11,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:11,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:19:11,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:19:15,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:17,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:21,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:19:21,243 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:19:21,243 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:19:21,244 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:19:21,518 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:19:22,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:23,519 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:19:25,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:26,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:19:26,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:19:28,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:30,234 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:19:33,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:35,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:38,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:41,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:19:41,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:19:44,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:45,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:49,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:54,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:19:55,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:19:56,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:19:56,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:20:00,236 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:20:00,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:05,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:05,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:10,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:11,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:20:11,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:20:15,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:16,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:21,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:25,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:26,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:26,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:20:26,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:20:30,238 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:20:32,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:35,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:37,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:41,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:20:41,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:20:43,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:45,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:48,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:53,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:20:55,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:20:56,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:20:56,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:20:59,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:00,241 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:21:04,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:05,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:09,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:11,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:21:11,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:21:15,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:15,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:20,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:22,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:21:22,900 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:21:22,900 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:21:22,901 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:21:23,568 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:21:25,570 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:21:25,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:26,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:26,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:21:26,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:21:30,244 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:21:32,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:35,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:37,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:41,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:21:41,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:21:43,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:45,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:48,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:53,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:21:55,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:21:56,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:21:56,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:21:59,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:00,246 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:22:04,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:05,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:09,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:11,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:22:11,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:22:14,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:15,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:19,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:25,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:25,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:26,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:22:26,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:22:30,249 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:22:31,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:35,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:36,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:41,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:41,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:22:41,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:22:45,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:47,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:52,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:22:55,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:22:56,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:22:56,866 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:22:58,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:00,255 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:23:03,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:05,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:08,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:11,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:23:11,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:23:14,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:15,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:19,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:24,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:24,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:23:24,625 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:23:24,625 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:23:24,626 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:23:24,628 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:23:25,629 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:23:25,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:26,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:23:26,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:23:30,254 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:23:30,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:35,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:35,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:41,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:41,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:23:41,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:23:45,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:47,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:52,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:23:55,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:23:56,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:23:56,867 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:23:58,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:00,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:24:03,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:05,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:08,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:11,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:24:11,868 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:24:14,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:15,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:19,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:24,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:25,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:26,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:24:26,868 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:24:30,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:24:30,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:35,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:35,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:41,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:41,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:24:41,868 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:24:45,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:46,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:52,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:24:55,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:24:56,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:24:56,868 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:24:58,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:00,261 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:25:03,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:05,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:08,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:11,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:25:11,869 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:25:14,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:15,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:19,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:24,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:25,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:26,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:25:26,869 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:25:27,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:25:27,643 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:25:27,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:25:27,644 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:25:27,684 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:25:29,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:29,685 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:25:30,264 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:25:35,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:35,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:41,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:41,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:25:41,870 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:25:45,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:46,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:52,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:25:55,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:25:56,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:25:56,870 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:25:58,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:00,266 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:26:03,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:05,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:08,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:11,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:26:11,870 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:26:14,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:15,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:19,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:24,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:25,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:26,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:26:26,870 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:26:30,268 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:26:30,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:35,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:35,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:40,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:41,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:26:41,870 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:26:45,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:46,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:52,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:26:55,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:26:56,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:26:56,871 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:26:58,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:00,271 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:27:03,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:05,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:08,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:11,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:27:11,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:27:14,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:15,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:19,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:25,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:25,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:26,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:27:26,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:27:29,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:27:29,459 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:27:29,459 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:27:29,460 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:27:29,734 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:27:30,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:27:31,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:31,735 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:27:35,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:36,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:41,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:41,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:27:41,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:27:45,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:47,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:53,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:27:55,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:27:56,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:27:56,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:27:59,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:00,275 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:28:04,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:05,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:09,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:11,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:28:11,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:28:15,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:15,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:21,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:25,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:26,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:26,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:28:26,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:28:30,279 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:28:32,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:35,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:37,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:41,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:28:41,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:28:43,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:45,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:48,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:53,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:28:55,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:28:56,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:28:56,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:28:59,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:00,279 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:29:04,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:05,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:09,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:11,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:29:11,901 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:29:15,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:15,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:20,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:25,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:25,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:26,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:29:26,901 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:29:30,283 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:29:31,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:29:31,223 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:29:31,223 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:29:31,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:31,225 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:29:31,786 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:29:33,787 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:29:35,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:36,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:41,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:41,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:29:41,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:29:45,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:47,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:53,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:29:55,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:29:56,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:29:56,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:29:59,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:00,285 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:30:04,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:05,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:09,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:11,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:30:11,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:30:14,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:15,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:19,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:25,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:25,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:26,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:30:26,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:30:30,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:30:31,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:35,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:36,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:41,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:30:41,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:30:42,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:45,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:47,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:52,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:30:55,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:30:56,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:30:56,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:30:58,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:00,290 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:31:03,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:05,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:08,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:11,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:31:11,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:31:14,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:15,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:19,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:25,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:25,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:26,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:31:26,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:31:30,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:31:31,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:34,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:31:34,421 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:31:34,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:31:34,423 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:31:34,835 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:31:35,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:35,837 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:31:36,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:41,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:31:41,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:31:42,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:45,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:47,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:53,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:31:55,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:31:56,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:31:56,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:31:59,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:00,299 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:32:04,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:05,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:09,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:11,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:32:11,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:32:15,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:15,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:21,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:25,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:26,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:26,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:32:26,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:32:30,297 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:32:32,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:35,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:37,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:41,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:32:41,906 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:32:43,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:45,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:48,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:54,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:32:55,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:32:56,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:32:56,906 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:33:00,301 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:33:00,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:05,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:05,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:10,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:11,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:33:11,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:33:15,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:16,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:21,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:25,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:26,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:33:26,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:33:27,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:30,302 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:33:32,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:35,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:36,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:33:36,048 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:33:36,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:33:36,049 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:33:36,883 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:33:37,883 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:33:38,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:41,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:33:41,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:33:43,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:45,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:48,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:53,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:33:55,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:33:56,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:33:56,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:33:59,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:00,304 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:34:04,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:05,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:09,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:11,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:34:11,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:34:14,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:15,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:19,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:24,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:25,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:26,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:34:26,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:34:30,307 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:34:30,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:35,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:35,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:41,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:41,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:34:41,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:34:45,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:46,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:52,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:34:55,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:34:56,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:34:56,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:34:58,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:00,309 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:35:03,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:05,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:08,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:11,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:35:11,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:35:14,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:15,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:19,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:24,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:25,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:26,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:35:26,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:35:29,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:30,313 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:35:34,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:35,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:37,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:35:37,656 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:35:37,656 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:35:37,657 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:35:37,953 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:35:39,954 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:35:40,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:41,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:35:41,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:35:45,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:46,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:52,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:35:55,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:35:56,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:35:56,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:35:58,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:00,314 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:36:03,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:05,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:08,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:11,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:36:11,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:36:14,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:15,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:19,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:24,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:25,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:26,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:36:26,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:36:29,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:30,320 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:36:35,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:35,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:40,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:41,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:36:41,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:36:45,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:46,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:52,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:36:55,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:36:56,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:36:56,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:36:58,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:00,319 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:37:03,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:05,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:08,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:11,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:37:11,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:37:14,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:15,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:19,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:25,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:25,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:26,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:37:26,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:37:30,322 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:37:31,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:35,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:36,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:40,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:37:40,881 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:37:40,882 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:37:40,882 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:37:41,010 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:37:41,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:37:41,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:37:42,010 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:37:42,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:45,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:47,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:53,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:37:55,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:37:56,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:37:56,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:37:59,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:00,323 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:38:04,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:05,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:09,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:11,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:38:11,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:38:14,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:15,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:20,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:25,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:25,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:26,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:38:26,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:38:30,325 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:38:31,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:35,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:36,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:41,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:38:41,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:38:42,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:45,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:47,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:53,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:38:55,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:38:56,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:38:56,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:38:59,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:00,328 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:39:04,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:05,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:09,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:11,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:39:11,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:39:15,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:15,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:20,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:25,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:26,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:26,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:39:26,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:39:30,330 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:39:32,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:35,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:37,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:41,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:39:41,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:39:42,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:39:42,636 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:39:42,637 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:39:42,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:42,638 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:39:43,066 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:39:44,066 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:39:45,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:48,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:54,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:39:55,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:39:56,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:39:56,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:39:59,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:00,332 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:40:04,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:05,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:09,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:11,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:40:11,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:40:15,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:15,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:20,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:25,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:26,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:26,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:40:26,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:40:30,334 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:40:32,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:35,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:37,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:41,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:40:41,913 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:40:43,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:45,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:48,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:53,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:40:55,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:40:56,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:40:56,913 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:40:59,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:00,337 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:41:04,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:05,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:09,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:11,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:41:11,913 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:41:15,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:15,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:20,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:25,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:25,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:26,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:41:26,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:41:30,343 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:41:31,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:35,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:36,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:41,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:41:41,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:41:42,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:44,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:41:44,423 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:41:44,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:41:44,424 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:41:45,121 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:41:45,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:46,121 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:41:47,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:53,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:41:55,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:41:56,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:41:56,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:41:59,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:00,341 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:42:04,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:05,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:09,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:11,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:42:11,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:42:14,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:15,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:20,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:25,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:25,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:26,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:42:26,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:42:30,345 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:42:31,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:35,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:36,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:41,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:42:41,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:42:42,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:45,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:47,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:53,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:42:55,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:42:56,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:42:56,914 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:42:59,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:00,346 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:43:04,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:05,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:10,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:11,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:43:11,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:43:15,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:16,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:21,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:25,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:26,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:43:26,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:43:27,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:30,348 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:43:32,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:35,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:37,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:41,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:43:41,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:43:43,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:45,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:47,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:43:47,667 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:43:47,667 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:43:47,668 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:43:48,175 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:43:48,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:50,176 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:43:53,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:43:55,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:43:56,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:43:56,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:43:59,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:00,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:44:04,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:05,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:10,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:11,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:44:11,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:44:15,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:16,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:21,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:25,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:26,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:44:26,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:44:27,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:30,353 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:44:32,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:35,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:37,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:41,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:44:41,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:44:43,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:45,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:48,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:53,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:44:55,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:44:56,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:44:56,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:44:59,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:00,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:45:04,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:05,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:10,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:11,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:45:11,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:45:15,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:16,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:21,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:25,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:26,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:45:26,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:45:27,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:30,358 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:45:32,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:35,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:37,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:41,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:45:41,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:45:43,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:45,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:48,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:49,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:45:49,383 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:45:49,383 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:45:49,384 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:45:50,229 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:45:52,230 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:45:54,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:45:55,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:45:56,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:45:56,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:46:00,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:00,364 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:46:05,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:05,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:10,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:11,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:46:11,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:46:15,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:15,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:21,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:25,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:26,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:46:26,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:46:27,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:30,363 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:46:32,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:35,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:38,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:41,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:46:41,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:46:44,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:45,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:49,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:54,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:46:55,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:46:56,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:46:56,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:47:00,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:00,367 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:47:05,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:05,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:10,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:11,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:47:11,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:47:15,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:16,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:21,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:25,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:26,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:47:26,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:47:27,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:30,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:47:32,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:35,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:38,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:41,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:47:41,936 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:47:44,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:45,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:49,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:50,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:47:50,921 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:47:50,921 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:47:50,922 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:47:51,287 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:47:52,288 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:47:54,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:47:55,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:47:56,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:47:56,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:48:00,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:00,370 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:48:05,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:05,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:10,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:11,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:48:11,936 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:48:15,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:16,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:22,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:25,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:26,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:48:26,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:48:27,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:30,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:48:33,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:35,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:38,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:41,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:48:41,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:48:43,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:45,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:49,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:55,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:48:55,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:48:56,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:48:56,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:49:00,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:00,375 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:49:05,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:06,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:11,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:49:11,937 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:49:12,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:15,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:18,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:23,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:25,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:27,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:49:27,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:49:28,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:30,377 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:49:34,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:35,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:40,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:42,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:49:42,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:49:45,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:45,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:51,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:54,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:49:54,015 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:49:54,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:49:54,016 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:49:54,357 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:49:55,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:49:56,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:49:56,359 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:49:57,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:49:57,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:50:00,380 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:50:01,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:05,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:07,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:12,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:12,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:50:12,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:50:15,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:17,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:23,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:25,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:27,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:50:27,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:50:28,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:30,386 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:50:34,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:35,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:39,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:42,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:50:42,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:50:44,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:45,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:50,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:55,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:50:56,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:50:57,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:50:57,966 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:51:00,384 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:51:01,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:05,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:07,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:12,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:12,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:51:12,965 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:51:15,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:18,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:23,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:25,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:27,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:51:27,966 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:51:28,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:30,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:51:34,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:35,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:40,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:42,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:51:42,966 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:51:45,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:45,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:51,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:55,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:51:55,735 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:51:55,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:51:55,736 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:51:55,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:51:56,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:51:57,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:51:57,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:51:57,966 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:51:58,416 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:52:00,390 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:52:03,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:05,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:09,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:12,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:52:12,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:52:14,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:15,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:20,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:25,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:26,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:27,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:52:27,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:52:30,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:52:31,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:35,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:37,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:42,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:42,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:52:42,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:52:45,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:48,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:53,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:52:55,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:52:57,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:52:57,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:52:58,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:00,395 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:53:04,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:05,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:10,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:12,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:53:12,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:53:15,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:15,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:21,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:25,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:27,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:27,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:53:27,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:53:30,397 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:53:33,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:35,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:39,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:42,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:53:42,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:53:44,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:45,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:50,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:55,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:53:56,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:53:57,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:53:57,431 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:53:57,431 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:53:57,432 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:53:57,472 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:53:57,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:53:57,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:54:00,399 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:54:00,473 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:54:01,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:05,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:06,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:11,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:12,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:54:12,968 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:54:15,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:17,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:23,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:25,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:27,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:54:27,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:54:28,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:30,402 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:54:34,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:35,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:40,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:42,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:54:42,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:54:45,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:45,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:51,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:55,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:54:57,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:54:57,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:54:57,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:55:00,405 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:55:02,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:05,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:08,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:12,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:55:12,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:55:13,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:15,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:19,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:24,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:25,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:27,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:55:27,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:55:29,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:30,411 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:55:35,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:35,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:41,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:42,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:55:42,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:55:45,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:47,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:53,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:55:55,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:55:57,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:55:57,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:55:58,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:00,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:56:00,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:56:00,697 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:56:00,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:56:00,700 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:56:01,530 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:56:01,531 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:56:04,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:05,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:10,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:12,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:56:12,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:56:15,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:15,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:21,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:25,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:27,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:27,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:56:27,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:56:30,412 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:56:32,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:35,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:38,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:42,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:56:42,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:56:43,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:45,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:49,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:55,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:56:55,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:56:57,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:56:57,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:57:00,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:00,415 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:57:05,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:06,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:11,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:12,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:57:12,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:57:15,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:17,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:23,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:25,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:27,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:57:27,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:57:28,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:30,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:57:34,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:35,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:40,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:42,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:57:42,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:57:45,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:45,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:51,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:55,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:57:56,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:57:57,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:57:57,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:58:00,419 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:58:01,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:02,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 13:58:02,279 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 13:58:02,279 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 13:58:02,280 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 13:58:02,585 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 13:58:03,586 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 13:58:05,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:07,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:12,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:12,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:58:12,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:58:15,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:18,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:23,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:25,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:27,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:58:27,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:58:28,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:30,421 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:58:34,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:35,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:39,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:42,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:58:42,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:58:44,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:45,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:50,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:55,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:58:55,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:58:57,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:58:57,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:59:00,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:00,424 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:59:05,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:06,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:11,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:12,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:59:12,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:59:15,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:16,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:22,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:25,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:27,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:59:27,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:59:28,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:30,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 13:59:34,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:35,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:40,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:42,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:59:42,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 13:59:45,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:45,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:51,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:55,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 13:59:56,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 13:59:57,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 13:59:57,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:00:00,431 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:00:02,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:03,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:00:03,969 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:00:03,969 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:00:03,970 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:00:04,645 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:00:05,646 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:00:05,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:08,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:12,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:00:12,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:00:13,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:15,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:19,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:24,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:25,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:27,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:00:27,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:00:29,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:30,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:00:35,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:35,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:41,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:42,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:00:42,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:00:45,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:46,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:51,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:55,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:00:57,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:00:57,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:00:57,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:01:00,435 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:01:03,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:05,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:09,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:12,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:01:12,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:01:14,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:15,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:20,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:25,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:26,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:27,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:01:27,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:01:30,436 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:01:31,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:35,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:37,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:42,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:42,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:01:42,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:01:45,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:47,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:01:55,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:01:57,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:01:57,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:01:58,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:00,439 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:02:04,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:05,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:02:05,674 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:02:05,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:02:05,675 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:02:05,700 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:02:05,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:07,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:02:10,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:12,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:02:12,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:02:15,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:15,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:21,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:25,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:27,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:27,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:02:27,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:02:30,442 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:02:32,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:35,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:38,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:42,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:02:42,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:02:43,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:45,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:49,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:54,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:02:55,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:02:57,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:02:57,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:02:59,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:00,444 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:03:05,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:05,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:11,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:12,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:03:12,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:03:15,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:17,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:22,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:25,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:27,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:03:27,989 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:03:28,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:30,445 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:03:34,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:35,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:40,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:42,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:03:42,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:03:45,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:45,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:51,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:55,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:03:56,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:03:57,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:03:57,989 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:04:00,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:04:02,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:05,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:08,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:09,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:04:09,450 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:04:09,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:04:09,452 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:04:09,767 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:04:11,768 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:04:12,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:04:12,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:04:13,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:15,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:19,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:24,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:25,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:27,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:04:27,989 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:04:29,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:30,533 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:04:35,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:35,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:41,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:42,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:04:42,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:04:45,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:47,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:53,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:04:55,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:04:57,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:04:57,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:04:58,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:00,453 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:05:04,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:05,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:10,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:12,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:05:12,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:05:15,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:15,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:21,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:25,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:27,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:27,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:05:27,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:05:30,457 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:05:33,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:35,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:39,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:42,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:05:42,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:05:44,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:45,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:50,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:55,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:05:56,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:05:57,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:05:57,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:06:00,458 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:06:01,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:05,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:07,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:10,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:06:10,994 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:06:10,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:06:10,995 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:06:11,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:06:11,830 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:06:12,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:06:12,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:06:13,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:15,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:19,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:24,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:25,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:27,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:06:27,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:06:29,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:30,460 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:06:35,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:35,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:41,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:42,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:06:42,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:06:45,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:46,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:51,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:55,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:06:57,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:06:57,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:06:57,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:07:00,463 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:07:02,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:05,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:08,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:12,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:07:12,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:07:13,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:15,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:19,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:24,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:25,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:27,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:07:27,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:07:29,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:30,466 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:07:35,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:35,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:41,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:42,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:07:42,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:07:45,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:46,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:51,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:55,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:07:57,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:07:57,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:07:57,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:08:00,468 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:08:02,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:05,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:08,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:12,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:08:12,812 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:08:12,812 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:08:12,813 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:08:12,881 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:08:13,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:08:13,005 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:08:13,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:13,881 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:08:15,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:19,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:24,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:25,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:28,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:08:28,005 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:08:29,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:30,471 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:08:35,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:35,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:41,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:43,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:08:43,005 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:08:45,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:47,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:52,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:55,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:08:57,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:08:58,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:08:58,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:09:00,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:09:03,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:05,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:09,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:13,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:09:13,006 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:09:14,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:15,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:20,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:25,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:25,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:28,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:09:28,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:09:30,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:30,475 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:09:35,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:36,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:41,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:43,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:09:43,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:09:45,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:47,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:53,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:09:55,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:09:58,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:09:58,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:09:58,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:00,479 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:10:04,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:05,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:10,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:13,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:10:13,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:10:15,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:15,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:16,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:10:16,221 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:10:16,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:10:16,222 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:10:16,941 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:10:17,942 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:10:20,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:25,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:26,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:28,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:10:28,007 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:10:30,480 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:10:31,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:35,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:37,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:42,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:43,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:10:43,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:10:45,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:48,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:53,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:10:55,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:10:58,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:10:58,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:10:58,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:00,482 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:11:04,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:05,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:09,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:13,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:11:13,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:11:14,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:15,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:20,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:25,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:26,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:28,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:11:28,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:11:30,485 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:11:32,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:35,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:38,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:43,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:11:43,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:11:43,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:45,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:49,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:54,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:11:55,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:11:58,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:11:58,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:11:59,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:00,487 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:12:05,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:05,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:10,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:13,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:12:13,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:12:15,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:15,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:18,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:12:18,203 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:12:18,203 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:12:18,204 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:12:19,009 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:12:20,009 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:12:20,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:25,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:26,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:28,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:12:28,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:12:30,489 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:12:31,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:35,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:37,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:43,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:12:43,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:12:43,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:45,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:49,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:55,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:12:55,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:12:58,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:12:58,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:13:00,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:00,491 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:13:05,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:06,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:12,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:13,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:13:13,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:13:15,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:18,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:24,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:25,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:28,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:13:28,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:13:29,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:30,497 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:13:35,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:35,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:41,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:43,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:13:43,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:13:45,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:47,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:52,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:13:55,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:13:58,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:13:58,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:13:58,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:00,536 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:14:04,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:05,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:10,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:13,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:14:13,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:14:15,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:15,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:20,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:14:20,144 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:14:20,144 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:14:20,145 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:14:21,081 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:14:21,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:22,081 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:14:25,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:27,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:28,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:14:28,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:14:30,501 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:14:32,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:35,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:38,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:43,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:14:43,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:14:43,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:45,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:49,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:54,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:14:55,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:14:58,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:14:58,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:15:00,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:00,505 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:15:05,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:05,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:11,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:13,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:15:13,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:15:15,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:16,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:21,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:25,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:27,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:28,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:15:28,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:15:30,506 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:15:33,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:35,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:39,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:43,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:15:43,012 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:15:44,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:45,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:50,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:55,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:15:56,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:15:58,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:15:58,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:16:00,508 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:16:01,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:05,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:06,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:11,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:13,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:16:13,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:16:15,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:17,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:22,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:23,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:16:23,082 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:16:23,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:16:23,084 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:16:23,136 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:16:24,137 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:16:25,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:28,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:16:28,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:16:28,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:30,510 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:16:34,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:35,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:40,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:43,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:16:43,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:16:45,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:46,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:51,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:55,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:16:57,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:16:58,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:16:58,016 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:17:00,513 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:17:03,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:05,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:09,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:13,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:17:13,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:17:14,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:15,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:20,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:25,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:26,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:28,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:17:28,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:17:30,515 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:17:31,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:35,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:37,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:42,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:43,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:17:43,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:17:45,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:47,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:53,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:17:55,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:17:58,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:17:58,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:17:58,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:00,517 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:18:04,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:05,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:10,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:13,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:18:13,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:18:15,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:15,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:21,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:24,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:18:24,633 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:18:24,634 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:18:24,634 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:18:25,188 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:18:25,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:26,189 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:18:27,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:28,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:18:28,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:18:30,523 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:18:32,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:35,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:37,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:42,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:43,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:18:43,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:18:45,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:48,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:53,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:18:55,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:18:58,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:18:58,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:18:58,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:00,522 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:19:04,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:05,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:09,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:13,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:19:13,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:19:14,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:15,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:20,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:25,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:26,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:28,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:19:28,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:19:30,526 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:19:31,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:35,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:37,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:42,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:43,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:19:43,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:19:45,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:47,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:53,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:19:55,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:19:58,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:19:58,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:19:58,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:00,527 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:20:04,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:05,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:10,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:13,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:20:13,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:20:15,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:15,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:21,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:25,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:26,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:26,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:20:26,250 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:20:26,251 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:20:26,251 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:20:27,240 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:20:28,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:20:28,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:20:28,241 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:20:30,529 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:20:31,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:35,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:37,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:43,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:20:43,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:20:43,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:45,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:49,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:55,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:20:55,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:20:58,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:20:58,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:21:00,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:00,531 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:21:05,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:06,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:11,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:13,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:21:13,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:21:15,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:17,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:23,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:25,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:28,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:21:28,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:21:28,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:30,533 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:21:34,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:35,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:40,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:43,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:21:43,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:21:45,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:46,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:51,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:55,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:21:56,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:21:58,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:21:58,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:22:00,536 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:22:02,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:05,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:08,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:13,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:22:13,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:22:13,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:15,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:19,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:25,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:25,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:28,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:22:28,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:22:29,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:22:29,451 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:22:29,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:22:29,452 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:22:30,290 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:22:30,290 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:22:30,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:30,538 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:22:35,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:36,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:41,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:43,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:22:43,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:22:45,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:47,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:52,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:22:55,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:22:58,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:22:58,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:22:58,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:00,547 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:23:04,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:05,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:09,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:13,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:23:13,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:23:14,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:15,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:20,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:25,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:25,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:28,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:23:28,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:23:30,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:30,543 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:23:35,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:36,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:41,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:43,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:23:43,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:23:45,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:46,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:51,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:55,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:23:57,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:23:58,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:23:58,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:24:00,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:24:02,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:05,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:08,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:13,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:24:13,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:24:14,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:15,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:19,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:25,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:25,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:28,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:24:28,051 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:24:30,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:30,550 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:24:31,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:24:31,226 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:24:31,226 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:24:31,227 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:24:31,360 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:24:32,361 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:24:35,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:35,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:41,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:43,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:24:43,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:24:45,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:47,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:53,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:24:55,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:24:58,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:24:58,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:24:58,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:00,553 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:25:04,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:05,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:10,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:13,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:25:13,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:25:15,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:16,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:21,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:25,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:27,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:28,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:25:28,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:25:30,554 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:25:32,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:35,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:38,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:43,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:25:43,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:25:43,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:45,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:49,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:54,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:25:55,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:25:58,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:25:58,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:25:59,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:00,555 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:26:05,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:05,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:11,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:13,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:26:13,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:26:15,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:17,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:22,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:25,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:28,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:26:28,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:26:28,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:30,557 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:26:32,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:26:32,983 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:26:32,983 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:26:32,984 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:26:33,428 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:26:34,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:34,428 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:26:35,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:39,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:43,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:26:43,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:26:44,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:45,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:50,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:55,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:26:55,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:26:58,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:26:58,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:27:00,560 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:27:00,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:05,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:06,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:11,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:13,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:27:13,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:27:15,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:16,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:21,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:25,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:27,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:28,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:27:28,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:27:30,566 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:27:32,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:35,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:37,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:43,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:27:43,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:27:43,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:45,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:48,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:53,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:27:55,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:27:58,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:27:58,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:27:59,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:00,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:28:04,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:05,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:09,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:13,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:28:13,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:28:14,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:15,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:20,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:25,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:25,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:28,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:28:28,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:28:30,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:30,568 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:28:35,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:36,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:28:36,209 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:28:36,209 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:28:36,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:36,211 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:28:36,485 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:28:38,486 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:28:41,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:43,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:28:43,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:28:45,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:46,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:51,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:55,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:28:57,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:28:58,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:28:58,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:29:00,569 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:29:02,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:05,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:08,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:13,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:29:13,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:29:14,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:15,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:19,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:24,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:25,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:28,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:29:28,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:29:30,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:30,573 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:29:35,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:35,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:41,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:43,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:29:43,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:29:45,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:46,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:52,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:29:55,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:29:58,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:29:58,060 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:29:58,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:00,575 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:30:04,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:05,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:10,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:13,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:30:13,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:30:15,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:15,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:21,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:25,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:26,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:28,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:30:28,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:30:30,577 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:30:31,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:35,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:37,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:37,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:30:37,888 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:30:37,889 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:30:37,889 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:30:38,544 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:30:38,545 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:30:42,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:43,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:30:43,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:30:45,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:48,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:53,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:30:55,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:30:58,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:30:58,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:30:59,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:00,579 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:31:04,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:05,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:10,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:13,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:31:13,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:31:15,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:15,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:21,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:25,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:26,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:28,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:31:28,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:31:30,582 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:31:31,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:35,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:37,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:42,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:43,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:31:43,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:31:45,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:48,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:53,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:31:55,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:31:58,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:31:58,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:31:58,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:00,731 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:32:04,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:05,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:10,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:13,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:32:13,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:32:15,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:15,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:21,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:25,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:27,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:28,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:32:28,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:32:30,589 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:32:32,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:35,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:38,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:39,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:32:39,402 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:32:39,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:32:39,405 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:32:39,598 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:32:40,599 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:32:43,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:32:43,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:32:44,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:45,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:49,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:54,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:32:55,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:32:58,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:32:58,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:32:59,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:00,589 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:33:05,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:05,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:10,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:13,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:33:13,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:33:15,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:15,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:21,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:25,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:27,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:28,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:33:28,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:33:30,593 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:33:32,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:35,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:38,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:43,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:33:43,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:33:43,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:45,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:49,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:54,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:33:55,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:33:58,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:33:58,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:33:59,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:00,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:34:05,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:05,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:11,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:13,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:34:13,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:34:15,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:17,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:23,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:25,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:28,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:34:28,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:34:29,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:30,596 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:34:34,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:35,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:39,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:42,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:34:42,770 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:34:42,770 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:34:42,771 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:34:43,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:34:43,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:34:43,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:34:44,657 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:34:45,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:45,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:51,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:55,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:34:57,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:34:58,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:34:58,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:35:00,599 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:35:03,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:05,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:08,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:13,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:35:13,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:35:13,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:15,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:19,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:24,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:25,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:28,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:35:28,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:35:30,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:30,601 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:35:35,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:35,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:40,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:43,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:35:43,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:35:45,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:46,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:51,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:55,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:35:56,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:35:58,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:35:58,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:36:00,604 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:36:02,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:05,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:08,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:13,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:36:13,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:36:13,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:15,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:19,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:24,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:25,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:28,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:36:28,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:36:29,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:30,606 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:36:35,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:35,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:41,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:43,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:36:43,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:36:44,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:36:44,346 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:36:44,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:36:44,348 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:36:44,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:36:45,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:46,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:36:47,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:52,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:36:55,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:36:58,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:36:58,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:36:58,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:00,609 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:37:04,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:05,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:10,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:13,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:37:13,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:37:15,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:15,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:21,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:25,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:26,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:28,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:37:28,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:37:30,612 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:37:31,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:35,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:37,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:42,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:43,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:37:43,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:37:45,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:48,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:53,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:37:55,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:37:58,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:37:58,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:37:59,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:00,615 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:38:04,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:05,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:10,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:13,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:38:13,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:38:15,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:15,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:21,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:25,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:27,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:28,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:38:28,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:38:30,617 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:38:32,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:35,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:38,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:43,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:38:43,080 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:38:43,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:45,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:38:45,971 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:38:45,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:38:45,973 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:38:45,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:46,785 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:38:46,786 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:38:49,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:54,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:38:55,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:38:58,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:38:58,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:39:00,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:00,620 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:39:05,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:05,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:11,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:13,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:39:13,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:39:15,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:16,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:21,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:25,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:27,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:28,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:39:28,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:39:30,622 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:39:32,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:35,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:38,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:43,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:39:43,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:39:44,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:45,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:49,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:54,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:39:55,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:39:58,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:39:58,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:39:59,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:00,627 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:40:05,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:05,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:11,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:13,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:40:13,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:40:15,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:17,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:23,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:25,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:28,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:40:28,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:40:28,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:30,628 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:40:33,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:35,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:39,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:43,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:40:43,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:40:45,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:45,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:47,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:40:47,651 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:40:47,651 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:40:47,652 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:40:47,846 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:40:48,847 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:40:50,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:55,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:40:56,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:40:58,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:40:58,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:41:00,631 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:41:01,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:05,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:07,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:12,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:13,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:41:13,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:41:15,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:17,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:22,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:25,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:28,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:41:28,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:41:28,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:30,637 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:41:33,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:35,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:38,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:43,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:41:43,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:41:44,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:45,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:49,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:55,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:41:55,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:41:58,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:41:58,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:42:00,636 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:42:01,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:05,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:06,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:11,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:13,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:42:13,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:42:15,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:16,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:21,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:25,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:27,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:28,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:42:28,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:42:30,640 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:42:32,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:35,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:37,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:42,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:43,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:42:43,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:42:45,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:48,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:50,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:42:50,730 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:42:50,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:42:50,731 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:42:50,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:42:52,901 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:42:53,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:42:55,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:42:58,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:42:58,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:42:59,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:00,641 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:43:04,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:05,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:09,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:13,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:43:13,088 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:43:15,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:15,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:20,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:25,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:26,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:28,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:43:28,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:43:30,643 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:43:32,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:35,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:37,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:43,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:43:43,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:43:43,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:45,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:48,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:53,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:43:55,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:43:58,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:43:58,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:43:59,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:00,648 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:44:04,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:05,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:10,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:13,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:44:13,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:44:15,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:16,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:21,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:25,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:26,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:28,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:44:28,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:44:30,650 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:44:31,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:35,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:37,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:42,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:43,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:44:43,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:44:45,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:47,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:52,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:44:52,287 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:44:52,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:44:52,288 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:44:52,947 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:44:53,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:44:54,948 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:44:56,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:44:58,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:44:58,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:44:59,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:00,653 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:45:04,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:06,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:09,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:13,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:45:13,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:45:15,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:16,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:20,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:25,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:26,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:28,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:45:28,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:45:30,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:45:31,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:36,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:36,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:41,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:43,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:45:43,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:45:46,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:47,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:52,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:45:56,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:45:58,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:45:58,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:45:58,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:00,660 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:46:03,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:06,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:09,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:13,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:46:13,091 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:46:15,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:16,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:20,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:26,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:26,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:28,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:46:28,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:46:30,660 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:46:32,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:36,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:38,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:43,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:46:43,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:46:44,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:46,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:49,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:53,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:46:53,963 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:46:53,963 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:46:53,966 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:46:54,004 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:46:55,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:46:55,005 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:46:56,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:46:58,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:46:58,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:47:00,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:00,663 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:47:05,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:06,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:11,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:13,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:47:13,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:47:16,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:17,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:23,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:26,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:28,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:47:28,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:47:29,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:30,667 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:47:34,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:36,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:40,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:43,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:47:43,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:47:46,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:46,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:51,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:56,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:47:57,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:47:58,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:47:58,093 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:48:00,667 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:48:02,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:06,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:08,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:13,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:48:13,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:48:14,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:16,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:19,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:24,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:26,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:28,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:48:28,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:48:30,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:30,669 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:48:35,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:36,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:41,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:43,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:48:43,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:48:46,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:47,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:53,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:48:56,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:48:57,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:48:57,482 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:48:57,483 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:48:57,484 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:48:58,073 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:48:58,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:48:58,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:48:59,074 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:48:59,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:00,673 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:49:04,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:06,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:09,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:13,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:49:13,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:49:15,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:16,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:20,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:26,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:26,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:28,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:49:28,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:49:30,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:49:31,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:36,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:36,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:42,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:43,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:49:43,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:49:46,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:47,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:52,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:56,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:49:57,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:49:58,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:49:58,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:50:00,677 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:50:03,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:06,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:09,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:13,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:50:13,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:50:15,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:16,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:20,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:25,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:26,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:28,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:50:28,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:50:30,683 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:50:31,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:36,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:36,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:42,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:43,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:50:43,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:50:46,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:48,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:54,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:50:56,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:50:58,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:50:58,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:50:59,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:50:59,089 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:50:59,089 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:50:59,090 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:50:59,139 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:51:00,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:00,681 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:51:01,141 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:51:05,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:06,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:10,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:13,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:51:13,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:51:16,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:16,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:21,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:26,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:27,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:28,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:51:28,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:51:30,686 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:51:32,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:36,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:38,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:43,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:51:43,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:51:44,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:46,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:49,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:55,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:51:56,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:51:58,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:51:58,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:52:00,686 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:52:01,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:06,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:06,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:11,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:13,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:52:13,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:52:16,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:16,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:22,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:26,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:27,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:28,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:52:28,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:52:30,689 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:52:32,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:36,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:38,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:43,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:52:43,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:52:44,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:46,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:49,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:54,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:52:56,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:52:58,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:52:58,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:53:00,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:00,692 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:53:00,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:53:00,822 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:53:00,822 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:53:00,822 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:53:01,206 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:53:03,207 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:53:05,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:06,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:10,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:13,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:53:13,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:53:16,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:21,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:26,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:27,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:28,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:53:28,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:53:30,694 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:53:32,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:36,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:38,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:43,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:53:43,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:53:44,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:46,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:49,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:54,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:53:56,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:53:58,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:53:58,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:54:00,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:00,696 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:54:05,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:06,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:10,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:13,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:54:13,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:54:16,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:16,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:21,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:26,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:27,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:28,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:54:28,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:54:30,698 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:54:33,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:36,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:39,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:43,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:54:43,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:54:45,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:46,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:50,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:55,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:54:56,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:54:58,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:54:58,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:55:00,705 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:55:01,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:04,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:55:04,261 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:55:04,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:55:04,263 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:55:05,258 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:55:05,259 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:55:06,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:06,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:12,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:13,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:55:13,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:55:16,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:18,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:23,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:26,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:28,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:55:28,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:55:29,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:30,704 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:55:34,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:36,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:39,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:43,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:55:43,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:55:45,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:46,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:50,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:55,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:55:56,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:55:58,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:55:58,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:56:00,705 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:56:01,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:06,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:06,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:12,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:13,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:56:13,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:56:16,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:17,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:22,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:26,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:28,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:56:28,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:56:28,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:30,708 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:56:33,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:36,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:38,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:43,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:56:43,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:56:44,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:46,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:49,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:54,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:56:56,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:56:58,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:56:58,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:57:00,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:00,710 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:57:05,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:06,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:06,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:57:06,036 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:57:06,036 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:57:06,037 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:57:06,315 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:57:07,316 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:57:10,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:13,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:57:13,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:57:16,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:16,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:21,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:26,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:26,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:28,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:57:28,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:57:30,715 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:57:31,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:36,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:37,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:42,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:43,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:57:43,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:57:46,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:48,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:53,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:57:56,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:57:58,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:57:58,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:57:59,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:00,717 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:58:04,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:06,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:10,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:13,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:58:13,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:58:16,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:16,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:21,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:26,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:27,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:28,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:58:28,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:58:30,719 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:58:32,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:36,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:38,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:43,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:58:43,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:58:44,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:46,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:49,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:54,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:58:56,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:58:58,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:58:58,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:59:00,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:00,721 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:59:05,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:06,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:07,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 14:59:07,736 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 14:59:07,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 14:59:07,737 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 14:59:08,369 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 14:59:09,369 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 14:59:11,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:13,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:59:13,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:59:16,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:16,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:21,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:26,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:27,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:28,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:59:28,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:59:30,724 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 14:59:32,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:36,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:37,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:42,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:43,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:59:43,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:59:46,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:48,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:53,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 14:59:56,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 14:59:58,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 14:59:58,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 14:59:59,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:00,730 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:00:04,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:06,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:10,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:13,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:00:13,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:00:16,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:16,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:21,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:26,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:27,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:28,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:00:28,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:00:30,729 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:00:32,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:36,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:38,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:43,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:00:43,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:00:44,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:46,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:49,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:55,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:00:56,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:00:58,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:00:58,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:01:00,733 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:01:01,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:06,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:06,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:10,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:01:10,806 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:01:10,806 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:01:10,807 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:01:11,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:01:11,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:13,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:01:13,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:01:13,416 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:01:16,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:16,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:21,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:26,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:27,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:28,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:01:28,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:01:30,734 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:01:32,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:36,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:38,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:43,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:01:43,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:01:44,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:46,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:49,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:55,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:01:56,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:01:58,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:01:58,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:02:00,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:02:01,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:06,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:06,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:11,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:13,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:02:13,115 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:02:16,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:17,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:22,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:26,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:28,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:02:28,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:02:28,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:30,739 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:02:33,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:36,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:39,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:43,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:02:43,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:02:45,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:46,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:50,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:55,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:02:56,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:02:58,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:02:58,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:03:00,741 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:03:01,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:06,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:06,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:12,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:03:12,519 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:03:12,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:03:12,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:12,521 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:03:13,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:03:13,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:03:13,467 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:03:15,468 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:03:16,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:17,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:22,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:26,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:28,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:03:28,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:03:28,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:30,744 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:03:33,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:36,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:38,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:43,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:03:43,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:03:44,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:46,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:49,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:54,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:03:56,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:03:58,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:03:58,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:04:00,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:00,746 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:04:05,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:06,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:11,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:13,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:04:13,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:04:16,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:17,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:22,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:26,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:28,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:04:28,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:04:28,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:30,753 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:04:33,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:36,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:39,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:43,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:04:43,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:04:45,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:46,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:50,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:55,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:04:56,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:04:58,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:04:58,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:05:00,752 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:05:01,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:06,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:06,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:11,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:13,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:05:13,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:05:14,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:05:14,068 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:05:14,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:05:14,070 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:05:14,515 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:05:15,516 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:05:16,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:17,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:22,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:26,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:28,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:05:28,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:05:28,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:30,754 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:05:33,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:36,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:39,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:43,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:05:43,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:05:45,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:46,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:50,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:56,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:05:56,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:05:58,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:05:58,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:06:00,758 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:06:01,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:06,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:06,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:12,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:13,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:06:13,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:06:16,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:18,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:23,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:26,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:28,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:06:28,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:06:29,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:30,759 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:06:34,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:36,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:40,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:43,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:06:43,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:06:46,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:46,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:51,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:56,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:06:57,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:06:58,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:06:58,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:07:00,761 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:07:02,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:06,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:08,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:13,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:07:13,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:07:14,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:16,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:17,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:07:17,297 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:07:17,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:07:17,298 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:07:17,568 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:07:19,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:19,570 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:07:24,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:26,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:28,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:07:28,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:07:30,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:30,764 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:07:35,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:36,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:41,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:43,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:07:43,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:07:46,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:47,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:53,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:07:56,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:07:58,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:07:58,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:07:59,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:00,767 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:08:04,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:06,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:10,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:13,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:08:13,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:08:16,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:16,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:21,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:26,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:27,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:28,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:08:28,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:08:30,769 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:08:32,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:36,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:38,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:43,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:08:43,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:08:44,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:46,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:49,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:55,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:08:56,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:08:58,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:08:58,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:09:00,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:09:01,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:06,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:06,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:12,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:13,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:09:13,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:09:16,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:18,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:19,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:09:19,179 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:09:19,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:09:19,180 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:09:19,628 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:09:21,629 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:09:23,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:26,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:28,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:09:28,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:09:29,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:30,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:09:34,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:36,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:40,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:43,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:09:43,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:09:46,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:46,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:51,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:56,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:09:56,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:09:58,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:09:58,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:10:00,778 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:10:02,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:06,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:08,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:13,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:10:13,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:10:14,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:16,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:19,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:25,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:26,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:28,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:10:28,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:10:30,778 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:10:31,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:36,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:36,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:41,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:43,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:10:43,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:10:46,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:47,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:52,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:10:56,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:10:58,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:10:58,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:10:58,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:00,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:11:03,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:06,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:09,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:13,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:11:13,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:11:15,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:16,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:20,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:11:20,983 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:11:20,984 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:11:20,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:20,985 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:11:21,680 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:11:23,682 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:11:26,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:26,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:28,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:11:28,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:11:30,783 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:11:31,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:36,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:37,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:43,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:11:43,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:11:43,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:46,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:48,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:54,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:11:56,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:11:58,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:11:58,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:12:00,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:00,785 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:12:05,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:06,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:11,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:13,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:12:13,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:12:16,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:17,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:22,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:26,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:27,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:28,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:12:28,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:12:30,788 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:12:33,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:36,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:39,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:43,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:12:43,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:12:45,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:46,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:50,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:55,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:12:56,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:12:58,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:12:58,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:13:00,791 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:13:01,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:06,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:06,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:11,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:13,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:13:13,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:13:16,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:16,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:22,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:13:22,833 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:13:22,834 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:13:22,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:22,835 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:13:23,735 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:13:25,736 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:13:26,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:27,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:28,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:13:28,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:13:30,798 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:13:33,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:36,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:39,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:43,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:13:43,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:13:45,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:46,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:50,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:56,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:13:56,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:13:58,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:13:58,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:14:00,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:14:02,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:06,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:07,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:13,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:14:13,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:14:13,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:16,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:18,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:23,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:26,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:28,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:14:28,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:14:29,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:30,800 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:14:34,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:36,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:40,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:43,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:14:43,146 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:14:46,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:46,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:51,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:56,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:14:57,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:14:58,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:14:58,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:15:00,800 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:15:02,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:06,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:08,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:13,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:15:13,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:15:14,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:16,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:19,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:24,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:26,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:15:26,183 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:15:26,183 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:15:26,184 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:15:26,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:26,788 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:15:27,788 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:15:28,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:15:28,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:15:30,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:30,803 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:15:35,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:36,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:40,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:43,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:15:43,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:15:46,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:46,455 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:51,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:56,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:15:57,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:15:58,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:15:58,147 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:16:00,805 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:16:02,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:06,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:08,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:13,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:16:13,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:16:14,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:16,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:19,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:25,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:26,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:28,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:16:28,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:16:30,807 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:16:31,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:36,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:36,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:41,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:43,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:16:43,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:16:46,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:47,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:52,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:56,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:16:57,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:16:58,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:16:58,148 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:17:00,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:17:03,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:06,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:09,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:13,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:17:13,149 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:17:15,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:16,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:21,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:26,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:27,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:27,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:17:27,956 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:17:27,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:17:27,957 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:17:28,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:17:28,158 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:17:28,844 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:17:29,844 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:17:30,812 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:17:32,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:36,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:37,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:42,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:43,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:17:43,158 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:17:46,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:47,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:53,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:17:56,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:17:58,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:17:58,158 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:17:58,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:00,817 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:18:04,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:06,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:10,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:13,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:18:13,159 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:18:15,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:16,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:21,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:26,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:26,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:28,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:18:28,159 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:18:30,817 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:18:31,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:36,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:37,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:42,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:43,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:18:43,159 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:18:46,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:48,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:53,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:18:56,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:18:58,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:18:58,159 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:18:58,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:00,819 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:19:04,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:06,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:10,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:13,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:19:13,159 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:19:15,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:16,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:21,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:26,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:26,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:28,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:19:28,160 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:19:29,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:19:29,839 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:19:29,839 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:19:29,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:19:29,900 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:19:30,822 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:19:31,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:31,901 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:19:36,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:37,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:42,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:43,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:19:43,160 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:19:46,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:48,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:53,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:56,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:19:58,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:19:58,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:19:58,160 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:20:00,824 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:20:03,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:07,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:08,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:13,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:20:13,161 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:20:14,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:17,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:19,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:24,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:27,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:28,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:20:28,161 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:20:30,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:30,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:20:35,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:37,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:41,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:43,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:20:43,161 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:20:47,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:47,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:52,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:20:57,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:20:58,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:20:58,161 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:20:58,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:00,829 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:21:03,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:07,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:09,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:13,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:21:13,161 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:21:15,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:17,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:20,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:25,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:27,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:28,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:21:28,165 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:21:30,831 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:21:30,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:33,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:21:33,068 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:21:33,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:21:33,070 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:21:33,956 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:21:35,957 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:21:36,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:37,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:41,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:43,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:21:43,165 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:21:47,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:47,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:52,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:21:57,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:21:58,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:21:58,165 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:21:58,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:00,834 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:22:03,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:07,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:09,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:13,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:22:13,165 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:22:15,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:17,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:20,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:26,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:27,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:28,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:22:28,166 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:22:30,836 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:22:31,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:37,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:37,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:42,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:43,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:22:43,166 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:22:47,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:48,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:53,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:22:57,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:22:58,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:22:58,168 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:22:59,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:00,842 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:23:04,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:07,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:10,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:13,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:23:13,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:23:16,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:17,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:21,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:27,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:27,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:28,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:23:28,168 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:23:30,840 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:23:32,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:34,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:23:34,759 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:23:34,759 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:23:34,761 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:23:35,009 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:23:36,010 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:23:37,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:38,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:43,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:23:43,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:23:44,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:47,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:49,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:54,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:23:57,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:23:58,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:23:58,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:24:00,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:00,843 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:24:05,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:07,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:10,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:13,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:24:13,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:24:16,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:17,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:21,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:26,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:27,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:28,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:24:28,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:24:30,847 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:24:31,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:37,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:37,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:42,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:43,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:24:43,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:24:47,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:48,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:53,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:24:57,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:24:58,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:24:58,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:24:59,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:00,847 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:25:04,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:07,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:10,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:13,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:25:13,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:25:16,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:17,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:21,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:26,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:27,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:28,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:25:28,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:25:30,850 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:25:31,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:36,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:25:36,503 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:25:36,503 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:25:36,504 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:25:37,060 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:25:37,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:37,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:38,060 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:25:42,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:43,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:25:43,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:25:47,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:48,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:53,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:25:57,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:25:58,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:25:58,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:25:59,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:00,852 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:26:04,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:07,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:10,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:13,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:26:13,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:26:16,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:17,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:21,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:27,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:27,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:28,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:26:28,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:26:30,855 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:26:32,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:37,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:38,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:43,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:26:43,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:26:44,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:47,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:49,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:55,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:26:57,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:26:58,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:26:58,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:27:00,858 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:27:00,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:06,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:07,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:12,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:13,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:27:13,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:27:17,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:18,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:23,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:27,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:28,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:27:28,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:27:29,321 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:30,863 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:27:34,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:37,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:39,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:27:39,889 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:27:39,890 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:27:39,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:39,891 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:27:40,112 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:27:42,113 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:27:43,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:27:43,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:27:45,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:47,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:50,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:56,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:27:57,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:27:58,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:27:58,189 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:28:00,862 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:28:01,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:07,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:07,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:12,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:13,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:28:13,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:28:17,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:18,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:23,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:27,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:28,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:28:28,191 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:28:29,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:30,866 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:28:34,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:37,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:40,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:43,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:28:43,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:28:46,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:47,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:51,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:56,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:28:57,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:28:58,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:28:58,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:29:00,867 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:29:01,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:07,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:07,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:12,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:13,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:29:13,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:29:17,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:18,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:23,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:27,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:28,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:29:28,190 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:29:29,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:30,868 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:29:34,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:37,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:40,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:41,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:29:41,447 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:29:41,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:29:41,448 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:29:42,160 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:29:43,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:29:43,191 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:29:44,161 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:29:46,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:47,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:51,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:56,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:29:57,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:29:58,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:29:58,192 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:30:00,871 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:30:01,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:07,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:07,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:12,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:13,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:30:13,192 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:30:17,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:18,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:23,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:27,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:28,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:30:28,192 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:30:29,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:30,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:30:34,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:37,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:39,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:43,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:30:43,192 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:30:45,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:47,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:50,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:55,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:30:57,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:30:58,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:30:58,192 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:31:00,876 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:31:00,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:06,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:07,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:12,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:13,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:31:13,193 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:31:17,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:18,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:23,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:27,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:28,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:31:28,193 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:31:29,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:30,879 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:31:34,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:37,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:40,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:43,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:31:43,162 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:31:43,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:31:43,163 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:31:43,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:31:43,203 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:31:43,213 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:31:45,214 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:31:46,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:47,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:51,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:56,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:31:57,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:31:58,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:31:58,194 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:32:00,886 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:32:01,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:07,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:07,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:12,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:13,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:32:13,194 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:32:17,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:18,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:23,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:27,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:28,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:32:28,194 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:32:29,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:30,883 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:32:34,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:37,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:40,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:43,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:32:43,195 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:32:46,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:47,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:51,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:56,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:32:57,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:32:58,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:32:58,211 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:33:00,887 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:33:01,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:07,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:07,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:12,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:13,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:33:13,211 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:33:17,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:18,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:24,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:27,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:28,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:33:28,211 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:33:30,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:30,889 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:33:35,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:37,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:41,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:43,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:33:43,211 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:33:46,458 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:33:46,459 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:33:46,460 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:33:46,461 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:33:47,262 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:33:47,262 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:33:47,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:47,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:52,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:33:57,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:33:58,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:33:58,212 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:33:58,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:00,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:34:03,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:07,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:09,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:13,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:34:13,212 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:34:15,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:17,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:20,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:25,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:27,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:28,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:34:28,212 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:34:30,893 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:34:30,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:36,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:37,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:42,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:43,211 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:34:43,212 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:34:47,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:48,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:53,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:34:57,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:34:58,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:34:58,212 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:34:59,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:00,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:35:04,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:07,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:10,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:13,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:35:13,213 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:35:16,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:17,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:21,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:26,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:27,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:28,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:35:28,213 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:35:30,897 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:35:31,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:37,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:37,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:42,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:43,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:35:43,213 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:35:47,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:48,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:35:48,151 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:35:48,152 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:35:48,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:48,153 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:35:48,314 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:35:49,315 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:35:53,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:35:57,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:35:58,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:35:58,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:35:59,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:00,900 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:36:04,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:07,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:10,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:13,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:36:13,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:36:16,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:17,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:21,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:26,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:27,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:28,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:36:28,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:36:30,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:36:31,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:37,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:37,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:42,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:43,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:36:43,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:36:47,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:48,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:53,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:36:57,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:36:58,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:36:58,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:36:59,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:00,904 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:37:04,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:07,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:10,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:13,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:37:13,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:37:16,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:17,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:21,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:26,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:27,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:28,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:37:28,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:37:30,908 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:37:31,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:37,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:37,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:42,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:43,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:37:43,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:37:47,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:48,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:50,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:37:50,045 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:37:50,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:37:50,046 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:37:50,370 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:37:51,371 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:37:53,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:37:57,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:37:58,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:37:58,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:37:59,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:00,908 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:38:04,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:07,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:10,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:13,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:38:13,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:38:16,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:17,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:21,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:26,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:27,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:28,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:38:28,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:38:30,911 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:38:31,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:37,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:37,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:42,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:43,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:38:43,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:38:47,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:48,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:53,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:38:57,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:38:58,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:38:58,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:38:59,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:00,914 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:39:04,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:07,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:10,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:13,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:39:13,218 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:39:16,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:17,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:21,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:26,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:27,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:29,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:39:29,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:39:30,916 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:39:31,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:37,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:37,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:42,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:44,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:39:44,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:39:47,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:47,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:52,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:53,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:39:53,220 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:39:53,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:39:53,222 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:39:53,425 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:39:55,426 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:39:57,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:39:58,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:39:59,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:39:59,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:40:00,919 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:40:03,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:07,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:09,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:14,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:40:14,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:40:14,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:17,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:20,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:25,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:27,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:29,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:40:29,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:40:30,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:30,920 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:40:36,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:37,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:42,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:44,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:40:44,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:40:47,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:48,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:53,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:57,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:40:59,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:40:59,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:40:59,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:41:00,923 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:41:04,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:07,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:10,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:14,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:41:14,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:41:15,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:17,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:21,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:26,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:27,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:29,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:41:29,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:41:30,930 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:41:31,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:37,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:37,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:42,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:44,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:41:44,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:41:47,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:48,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:53,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:54,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:41:54,955 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:41:54,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:41:54,956 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:41:55,476 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:41:55,476 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:41:57,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:41:59,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:41:59,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:41:59,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:42:00,928 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:42:04,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:07,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:10,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:14,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:42:14,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:42:15,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:17,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:21,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:26,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:27,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:29,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:42:29,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:42:30,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:42:31,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:37,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:37,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:42,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:44,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:42:44,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:42:47,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:48,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:53,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:57,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:42:59,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:42:59,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:42:59,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:43:00,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:43:04,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:07,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:10,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:14,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:43:14,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:43:15,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:17,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:21,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:26,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:27,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:29,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:43:29,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:43:30,935 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:43:31,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:37,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:37,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:42,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:44,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:43:44,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:43:47,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:48,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:53,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:56,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:43:56,820 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:43:56,821 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:43:56,821 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:43:57,540 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:43:57,540 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:43:57,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:43:58,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:43:59,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:43:59,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:44:00,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:44:03,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:07,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:09,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:14,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:14,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:44:14,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:44:17,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:20,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:25,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:27,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:29,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:44:29,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:44:30,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:30,939 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:44:36,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:37,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:41,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:44,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:44:44,692 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:44:46,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:47,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:52,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:57,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:44:58,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:44:59,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:44:59,692 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:45:00,942 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:45:03,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:07,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:09,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:14,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:14,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:45:14,692 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:45:17,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:20,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:26,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:27,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:29,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:45:29,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:45:30,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:45:31,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:37,567 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:37,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:42,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:44,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:45:44,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:45:47,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:48,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:53,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:57,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:45:59,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:45:59,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:45:59,693 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:46:00,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:46:00,149 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:46:00,149 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:46:00,150 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:46:00,608 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:46:00,952 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:46:01,609 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:46:04,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:07,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:10,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:14,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:46:14,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:46:15,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:17,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:21,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:27,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:27,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:29,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:46:29,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:46:30,949 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:46:32,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:37,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:38,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:43,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:44,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:46:44,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:46:47,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:49,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:54,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:46:57,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:46:59,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:46:59,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:46:59,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:00,953 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:47:05,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:07,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:11,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:14,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:47:14,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:47:16,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:17,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:22,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:27,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:28,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:29,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:47:29,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:47:30,954 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:47:33,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:37,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:39,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:44,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:47:44,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:47:44,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:47,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:50,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:55,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:47:57,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:47:59,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:47:59,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:48:00,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:00,956 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:48:01,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:48:01,728 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:48:01,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:48:01,730 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:48:02,669 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:48:03,669 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:48:06,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:07,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:12,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:14,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:48:14,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:48:17,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:17,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:22,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:27,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:28,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:29,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:48:29,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:48:30,959 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:48:33,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:37,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:39,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:44,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:44,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:48:44,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:48:47,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:50,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:56,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:48:57,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:48:59,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:48:59,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:49:00,962 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:49:01,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:07,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:07,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:12,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:14,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:49:14,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:49:17,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:18,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:23,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:27,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:29,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:29,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:49:29,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:49:30,964 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:49:34,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:37,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:39,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:44,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:49:44,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:49:44,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:47,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:50,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:56,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:49:57,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:49:59,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:49:59,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:50:00,966 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:50:01,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:03,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:50:03,578 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:50:03,578 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:50:03,579 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:50:03,721 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:50:05,722 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:50:07,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:07,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:12,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:14,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:50:14,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:50:17,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:17,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:23,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:27,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:29,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:29,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:50:29,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:50:30,971 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:50:34,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:37,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:40,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:44,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:50:44,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:50:45,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:47,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:51,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:56,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:50:57,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:50:59,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:50:59,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:51:00,971 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:51:01,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:07,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:07,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:12,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:14,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:51:14,701 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:51:17,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:17,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:22,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:27,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:28,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:29,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:51:29,701 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:51:31,038 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:51:33,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:37,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:39,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:44,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:44,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:51:44,701 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:51:47,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:50,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:55,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:51:57,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:51:59,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:51:59,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:52:00,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:01,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:52:05,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:52:05,507 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:52:05,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:52:05,509 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:52:05,768 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:52:06,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:07,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:07,769 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:52:11,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:14,701 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:52:14,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:52:16,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:17,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:22,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:27,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:29,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:29,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:52:29,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:52:31,043 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:52:33,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:38,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:39,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:44,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:44,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:52:44,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:52:49,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:49,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:55,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:52:59,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:52:59,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:52:59,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:53:00,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:01,045 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:53:06,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:09,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:11,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:14,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:53:14,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:53:16,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:19,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:22,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:28,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:29,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:29,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:53:29,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:53:31,047 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:53:34,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:39,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:39,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:44,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:53:44,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:53:44,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:49,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:50,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:55,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:53:59,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:53:59,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:53:59,706 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:54:00,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:01,049 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:54:06,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:08,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:54:08,844 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:54:08,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:54:08,846 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:54:09,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:09,831 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:54:09,831 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:54:12,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:14,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:54:14,706 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:54:17,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:19,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:23,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:28,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:29,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:29,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:54:29,706 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:54:31,052 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:54:34,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:39,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:39,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:44,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:54:44,707 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:54:44,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:49,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:50,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:56,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:54:59,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:54:59,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:54:59,707 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:55:01,060 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:55:02,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:07,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:09,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:12,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:14,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:55:14,708 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:55:17,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:19,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:23,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:28,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:29,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:29,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:55:29,708 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:55:31,057 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:55:34,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:39,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:39,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:44,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:55:44,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:55:44,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:49,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:50,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:56,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:55:59,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:55:59,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:55:59,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:56:01,061 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:56:02,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:07,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:09,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:10,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:56:10,618 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:56:10,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:56:10,619 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:56:10,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:56:11,887 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:56:12,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:14,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:56:14,710 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:56:17,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:19,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:23,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:29,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:29,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:29,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:56:29,711 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:56:31,062 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:56:35,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:39,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:41,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:44,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:56:44,711 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:56:46,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:49,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:52,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:58,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:56:59,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:56:59,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:56:59,712 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:57:01,064 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:57:04,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:09,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:09,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:14,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:14,716 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:57:14,717 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:57:19,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:20,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:25,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:29,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:29,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:57:29,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:57:30,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:31,066 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:57:36,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:39,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:42,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:44,717 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:57:44,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:57:47,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:49,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:53,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:58,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:57:59,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:57:59,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:57:59,720 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:58:01,069 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:58:04,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:09,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:09,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:12,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 15:58:12,335 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 15:58:12,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 15:58:12,336 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 15:58:12,929 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 15:58:13,929 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 15:58:14,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:58:14,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:58:14,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:19,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:20,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:25,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:29,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:29,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:58:29,718 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:58:30,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:31,071 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:58:36,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:39,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:42,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:44,718 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:58:44,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:58:47,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:49,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:53,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:58,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:58:59,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:58:59,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:58:59,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:59:01,073 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:59:04,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:09,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:09,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:14,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:14,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:59:14,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:59:19,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:20,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:25,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:29,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:29,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:59:29,719 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:59:30,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:31,079 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 15:59:36,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:39,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:41,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:44,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:59:44,720 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 15:59:46,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:49,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:52,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:57,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 15:59:59,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 15:59:59,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 15:59:59,720 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:00:01,078 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:00:03,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:08,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:09,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:14,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:14,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:00:14,720 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:00:15,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:00:15,583 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:00:15,584 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:00:15,586 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:00:15,990 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:00:17,990 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:00:19,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:20,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:25,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:29,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:29,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:00:29,721 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:00:30,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:31,080 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:00:36,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:39,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:42,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:44,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:00:44,721 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:00:47,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:49,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:53,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:58,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:00:59,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:00:59,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:00:59,721 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:01:01,084 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:01:04,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:09,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:09,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:14,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:14,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:01:14,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:01:19,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:20,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:25,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:29,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:29,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:01:29,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:01:30,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:31,085 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:01:36,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:39,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:42,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:44,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:01:44,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:01:47,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:49,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:53,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:58,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:01:59,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:01:59,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:01:59,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:02:01,087 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:02:04,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:09,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:09,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:14,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:14,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:02:14,722 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:02:17,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:02:17,500 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:02:17,500 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:02:17,501 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:02:18,046 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:02:18,046 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:02:19,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:20,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:25,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:29,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:29,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:02:29,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:02:30,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:31,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:02:36,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:39,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:42,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:44,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:02:44,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:02:47,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:49,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:53,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:58,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:02:59,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:02:59,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:02:59,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:03:01,092 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:03:04,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:09,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:09,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:14,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:14,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:03:14,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:03:19,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:20,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:25,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:29,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:29,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:03:29,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:03:30,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:31,095 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:03:36,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:39,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:41,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:44,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:03:44,724 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:03:46,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:49,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:52,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:57,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:03:59,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:03:59,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:03:59,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:04:01,097 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:04:03,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:08,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:09,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:14,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:14,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:04:14,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:04:19,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:19,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:04:19,353 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:04:19,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:04:19,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:19,355 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:04:20,099 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:04:20,100 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:04:25,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:29,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:29,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:04:29,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:04:30,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:31,103 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:04:36,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:39,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:42,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:44,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:04:44,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:04:47,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:49,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:53,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:58,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:04:59,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:04:59,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:04:59,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:05:01,101 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:05:04,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:09,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:09,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:14,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:14,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:05:14,725 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:05:19,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:20,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:25,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:29,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:29,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:05:29,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:05:30,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:31,105 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:05:36,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:39,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:42,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:44,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:05:44,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:05:47,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:49,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:53,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:59,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:05:59,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:05:59,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:05:59,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:06:01,106 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:06:04,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:09,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:09,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:14,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:06:14,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:06:14,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:19,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:20,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:22,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:06:22,586 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:06:22,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:06:22,587 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:06:23,161 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:06:24,162 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:06:25,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:29,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:29,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:06:29,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:06:30,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:31,108 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:06:36,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:39,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:42,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:44,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:06:44,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:06:47,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:49,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:53,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:58,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:06:59,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:06:59,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:06:59,727 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:07:01,111 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:07:04,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:09,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:09,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:14,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:14,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:07:14,728 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:07:19,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:20,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:26,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:29,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:29,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:07:29,728 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:07:31,113 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:07:32,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:37,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:39,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:43,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:44,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:07:44,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:07:48,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:49,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:54,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:07:59,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:07:59,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:07:59,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:07:59,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:01,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:08:05,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:09,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:11,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:14,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:08:14,729 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:08:16,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:19,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:22,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:24,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:08:24,292 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:08:24,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:08:24,294 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:08:25,232 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:08:26,233 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:08:27,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:29,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:29,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:08:29,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:08:31,118 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:08:33,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:38,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:39,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:44,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:44,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:08:44,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:08:49,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:50,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:55,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:08:59,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:08:59,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:08:59,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:09:00,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:01,123 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:09:06,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:09,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:11,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:14,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:09:14,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:09:16,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:19,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:22,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:27,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:29,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:29,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:09:29,730 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:09:31,122 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:09:33,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:38,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:39,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:44,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:44,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:09:44,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:09:49,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:50,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:55,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:09:59,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:09:59,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:09:59,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:10:00,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:01,126 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:10:06,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:09,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:12,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:14,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:10:14,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:10:17,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:19,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:23,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:25,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:10:25,839 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:10:25,839 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:10:25,840 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:10:26,300 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:10:28,301 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:10:28,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:29,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:29,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:10:29,731 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:10:31,127 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:10:34,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:39,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:39,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:44,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:44,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:10:44,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:10:49,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:50,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:55,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:10:59,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:10:59,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:10:59,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:11:00,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:01,129 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:11:06,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:09,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:12,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:14,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:11:14,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:11:17,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:19,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:23,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:28,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:29,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:29,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:11:29,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:11:31,132 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:11:34,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:39,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:39,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:44,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:44,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:11:44,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:11:49,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:50,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:55,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:11:59,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:11:59,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:11:59,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:12:00,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:01,136 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:12:06,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:09,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:12,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:14,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:12:14,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:12:17,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:19,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:23,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:28,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:12:28,906 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:12:28,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:12:28,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:28,908 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:12:29,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:29,354 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:12:29,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:12:29,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:12:30,354 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:12:31,136 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:12:34,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:39,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:39,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:44,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:44,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:12:44,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:12:49,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:49,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:54,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:59,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:12:59,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:12:59,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:12:59,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:13:01,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:13:04,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:09,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:10,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:14,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:13:14,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:13:15,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:19,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:21,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:26,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:29,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:29,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:13:29,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:13:31,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:13:32,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:37,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:39,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:43,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:44,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:13:44,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:13:48,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:49,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:54,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:13:59,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:13:59,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:13:59,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:13:59,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:01,145 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:14:05,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:09,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:11,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:14,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:14:14,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:14:16,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:19,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:22,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:27,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:29,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:29,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:14:29,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:14:30,571 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:14:30,573 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:14:30,574 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:14:30,575 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:14:31,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:14:31,412 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:14:32,414 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:14:33,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:38,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:39,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:44,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:44,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:14:44,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:14:49,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:50,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:55,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:14:59,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:14:59,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:14:59,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:15:00,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:01,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:15:06,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:09,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:12,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:14,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:15:14,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:15:17,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:19,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:23,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:28,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:29,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:29,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:15:29,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:15:31,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:15:34,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:39,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:39,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:44,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:44,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:15:44,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:15:49,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:50,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:55,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:15:59,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:15:59,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:15:59,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:16:00,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:01,154 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:16:06,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:09,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:12,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:14,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:16:14,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:16:17,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:19,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:23,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:29,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:29,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:29,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:16:29,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:16:31,157 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:16:32,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:16:32,347 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:16:32,347 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:16:32,348 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:16:32,471 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:16:34,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:34,473 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:16:39,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:40,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:44,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:16:44,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:16:45,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:49,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:51,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:56,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:16:59,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:16:59,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:16:59,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:17:01,159 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:17:02,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:07,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:09,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:13,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:14,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:17:14,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:17:18,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:19,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:24,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:29,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:29,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:17:29,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:17:29,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:31,163 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:17:35,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:39,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:40,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:44,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:17:44,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:17:45,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:49,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:51,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:56,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:17:59,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:17:59,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:17:59,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:18:01,168 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:18:02,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:07,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:09,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:18:13,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:15,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:18:15,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:18:19,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:19,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:18:24,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:29,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:18:30,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:30,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:18:30,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:18:31,167 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:18:35,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:35,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:18:35,611 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:18:35,611 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:18:35,614 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:18:36,530 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:18:36,530 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:18:39,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:18:40,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:45,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:18:45,970 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:18:46,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:49,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:18:51,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:56,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:18:59,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:00,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:19:00,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:19:01,169 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:19:02,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:07,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:09,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:13,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:15,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:19:15,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:19:19,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:19,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:24,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:29,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:30,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:30,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:19:30,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:19:31,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:19:35,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:39,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:41,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:45,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:19:45,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:19:47,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:49,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:19:52,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:58,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:19:59,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:00,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:20:00,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:20:01,175 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:20:04,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:09,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:09,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:14,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:15,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:20:15,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:20:19,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:20,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:25,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:29,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:30,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:20:30,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:20:31,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:31,177 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:20:36,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:37,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:20:37,276 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:20:37,277 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:20:37,278 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:20:37,602 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:20:38,602 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:20:39,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:42,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:45,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:20:45,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:20:48,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:49,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:20:53,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:58,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:20:59,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:00,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:21:00,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:21:01,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:21:04,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:09,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:09,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:14,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:15,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:21:15,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:21:19,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:19,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:24,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:29,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:30,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:30,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:21:30,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:21:31,182 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:21:35,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:39,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:41,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:45,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:21:45,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:21:47,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:49,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:21:52,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:57,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:21:59,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:00,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:22:00,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:22:01,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:22:03,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:08,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:09,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:14,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:15,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:22:15,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:22:19,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:20,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:25,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:29,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:30,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:22:30,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:22:31,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:31,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:22:36,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:38,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:22:38,508 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:22:38,508 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:22:38,509 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:22:38,655 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:22:39,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:40,655 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:22:42,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:45,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:22:45,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:22:48,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:49,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:22:53,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:58,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:22:59,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:00,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:23:00,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:23:01,192 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:23:04,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:09,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:09,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:14,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:15,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:23:15,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:23:19,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:20,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:25,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:29,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:30,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:23:30,974 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:23:31,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:31,192 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:23:36,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:39,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:42,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:45,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:23:45,977 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:23:48,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:49,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:23:53,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:58,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:23:59,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:00,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:24:00,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:24:01,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:24:04,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:09,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:09,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:15,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:15,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:24:15,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:24:19,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:21,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:27,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:29,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:30,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:24:30,978 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:24:31,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:24:33,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:38,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:39,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:41,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:24:41,653 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:24:41,653 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:24:41,655 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:24:41,712 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:24:42,714 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:24:43,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:45,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:24:45,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:24:49,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:49,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:24:54,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:24:59,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:00,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:00,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:25:00,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:25:01,199 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:25:06,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:09,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:11,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:15,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:25:15,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:25:17,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:19,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:22,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:28,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:29,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:30,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:25:30,979 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:25:31,201 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:25:34,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:39,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:39,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:44,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:45,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:25:45,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:25:49,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:25:49,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:55,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:25:59,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:00,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:26:00,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:26:01,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:01,203 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:26:06,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:09,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:11,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:15,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:26:15,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:26:17,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:19,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:22,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:27,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:29,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:30,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:26:30,982 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:26:31,208 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:26:33,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:38,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:39,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:43,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:26:43,517 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:26:43,517 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:26:43,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:43,518 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:26:43,779 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:26:44,779 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:26:45,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:26:45,984 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:26:49,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:26:51,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:26:52,784 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:26:54,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:00,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:00,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:27:00,982 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:27:01,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:27:01,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:02,789 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:27:05,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:10,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:11,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:15,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:27:15,982 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:27:16,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:21,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:21,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:26,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:30,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:27:30,983 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:27:31,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:34,559 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:27:34,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:40,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:41,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:45,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:27:45,983 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:27:46,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:51,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:27:51,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:27:56,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:00,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:28:00,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:28:01,297 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:28:01,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:02,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:07,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:11,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:12,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:15,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:28:15,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:28:17,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:21,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:23,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:28,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:30,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:28:30,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:28:31,379 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:28:31,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:34,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:39,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:41,504 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:45,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:45,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:28:45,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:28:50,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:28:51,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:28:56,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:00,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:29:00,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:29:01,218 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:29:01,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:02,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:29:02,516 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:29:02,516 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:29:02,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:02,519 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:29:02,876 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:29:04,877 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:29:08,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:11,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:14,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:15,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:29:15,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:29:19,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:21,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:25,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:30,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:29:30,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:29:31,262 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:29:31,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:31,505 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:36,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:41,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:41,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:45,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:29:45,989 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:29:46,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:51,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:29:52,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:29:58,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:00,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:30:00,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:30:01,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:04,747 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:30:04,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:10,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:11,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:15,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:30:15,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:30:16,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:21,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:21,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:26,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:30,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:30:30,990 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:30:31,251 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:30:31,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:31,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:37,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:41,507 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:42,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:45,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:30:45,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:30:47,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:51,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:30:52,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:30:58,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:00,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:31:00,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:31:01,228 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:31:01,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:03,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:04,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:31:04,559 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:31:04,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:31:04,561 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:31:04,942 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:31:06,943 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:31:09,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:11,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:14,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:15,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:31:15,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:31:19,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:21,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:25,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:30,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:31:30,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:31:31,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:31,584 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:31:31,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:37,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:41,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:43,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:45,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:31:45,991 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:31:48,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:51,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:31:54,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:31:59,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:00,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:32:00,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:32:01,239 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:32:01,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:05,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:11,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:11,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:15,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:32:15,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:32:16,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:21,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:22,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:28,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:30,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:32:30,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:32:31,243 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:32:31,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:33,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:38,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:41,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:44,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:45,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:32:45,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:32:49,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:32:51,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:32:54,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:00,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:00,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:33:00,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:33:01,237 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:33:01,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:05,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:07,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:33:07,939 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:33:07,940 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:33:07,942 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:33:08,026 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:33:10,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:11,030 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:33:11,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:15,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:33:15,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:33:16,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:21,511 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:21,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:27,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:30,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:33:30,993 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:33:31,240 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:33:31,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:32,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:37,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:41,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:42,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:45,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:33:45,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:33:48,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:51,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:33:53,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:33:58,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:00,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:34:00,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:34:01,244 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:34:01,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:04,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:10,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:11,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:15,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:34:15,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:34:16,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:21,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:21,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:26,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:30,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:34:30,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:34:31,250 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:34:31,513 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:31,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:36,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:41,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:42,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:45,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:34:45,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:34:47,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:51,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:34:53,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:34:58,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:00,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:35:00,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:35:01,248 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:35:01,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:04,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:09,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:10,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:35:10,106 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:35:10,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:35:10,108 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:35:11,097 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:35:11,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:13,098 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:35:15,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:15,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:35:15,995 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:35:20,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:21,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:26,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:31,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:35:31,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:35:31,251 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:35:31,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:31,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:37,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:41,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:43,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:46,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:35:46,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:35:49,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:35:51,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:35:54,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:00,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:01,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:36:01,008 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:36:01,265 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:36:01,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:06,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:11,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:12,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:16,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:36:16,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:36:17,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:21,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:22,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:28,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:31,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:36:31,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:36:31,281 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:36:31,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:33,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:39,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:41,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:45,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:46,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:36:46,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:36:51,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:36:51,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:36:56,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:01,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:37:01,009 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:37:01,265 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:37:01,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:02,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:07,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:11,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:12,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:37:12,235 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:37:12,235 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:37:12,237 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:37:13,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:37:13,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:15,158 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:37:16,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:37:16,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:37:18,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:21,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:23,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:28,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:31,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:37:31,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:37:31,261 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:37:31,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:33,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:39,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:41,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:44,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:46,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:37:46,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:37:49,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:37:52,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:37:54,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:00,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:01,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:38:01,010 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:38:01,267 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:38:02,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:05,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:11,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:12,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:16,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:38:16,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:38:17,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:22,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:22,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:27,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:31,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:38:31,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:38:31,279 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:38:32,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:32,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:38,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:42,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:44,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:46,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:38:46,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:38:49,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:38:52,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:38:55,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:00,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:01,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:39:01,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:39:01,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:39:02,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:05,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:11,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:12,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:16,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:39:16,011 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:39:17,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:22,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:22,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:25,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:39:25,390 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:39:25,390 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:39:25,391 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:39:26,222 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:39:27,222 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:39:27,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:31,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:39:31,013 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:39:31,273 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:39:32,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:33,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:38,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:42,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:43,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:46,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:39:46,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:39:48,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:52,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:39:54,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:39:59,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:01,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:40:01,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:40:01,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:40:02,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:05,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:10,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:12,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:16,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:40:16,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:40:16,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:21,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:22,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:27,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:31,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:40:31,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:40:31,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:40:32,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:33,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:38,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:42,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:44,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:46,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:40:46,014 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:40:49,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:40:52,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:40:55,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:00,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:01,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:41:01,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:41:01,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:41:02,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:06,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:11,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:12,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:16,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:41:16,015 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:41:17,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:22,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:22,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:27,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:41:27,295 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:41:27,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:41:27,297 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:41:27,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:28,277 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:41:29,278 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:41:31,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:41:31,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:41:31,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:41:32,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:33,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:38,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:42,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:44,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:46,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:41:46,026 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:41:49,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:41:52,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:41:54,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:00,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:01,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:42:01,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:42:01,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:42:02,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:06,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:12,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:12,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:16,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:42:16,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:42:17,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:22,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:23,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:28,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:31,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:42:31,030 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:42:31,287 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:42:32,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:34,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:40,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:42,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:46,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:42:46,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:42:46,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:51,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:42:52,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:42:57,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:01,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:43:01,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:43:01,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:43:02,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:03,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:08,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:12,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:14,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:16,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:43:16,031 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:43:19,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:22,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:24,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:29,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:43:29,396 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:43:29,397 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:43:29,397 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:43:30,349 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:43:30,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:31,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:43:31,032 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:43:31,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:43:31,350 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:43:32,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:35,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:40,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:42,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:45,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:46,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:43:46,033 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:43:51,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:43:52,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:43:56,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:01,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:44:01,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:44:01,295 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:44:02,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:02,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:07,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:12,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:12,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:16,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:44:16,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:44:17,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:22,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:23,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:29,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:31,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:44:31,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:44:31,297 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:44:32,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:34,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:40,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:42,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:45,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:46,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:44:46,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:44:51,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:44:52,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:44:57,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:01,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:45:01,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:45:01,299 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:45:02,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:02,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:07,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:12,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:13,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:16,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:45:16,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:45:18,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:22,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:24,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:29,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:31,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:45:31,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:45:31,302 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:45:32,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:33,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:45:33,587 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:45:33,587 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:45:33,588 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:45:34,436 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:45:34,437 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:45:35,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:40,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:42,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:45,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:46,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:45:46,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:45:51,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:45:52,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:45:56,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:01,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:46:01,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:46:01,308 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:46:02,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:02,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:07,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:12,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:13,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:16,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:46:16,040 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:46:19,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:22,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:25,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:31,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:46:31,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:46:31,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:31,306 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:46:32,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:36,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:41,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:42,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:46,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:46:46,042 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:46:47,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:52,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:46:52,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:46:57,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:01,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:47:01,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:47:01,310 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:47:02,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:02,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:08,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:12,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:14,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:16,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:47:16,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:47:19,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:22,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:24,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:29,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:31,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:47:31,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:47:31,311 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:47:32,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:35,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:47:35,332 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:47:35,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:47:35,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:35,334 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:47:35,496 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:47:36,497 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:47:40,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:42,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:46,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:47:46,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:47:46,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:51,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:47:52,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:47:57,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:01,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:48:01,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:48:01,313 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:48:02,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:03,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:09,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:12,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:15,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:16,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:48:16,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:48:21,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:22,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:27,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:31,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:48:31,056 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:48:31,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:48:32,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:32,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:37,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:42,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:43,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:46,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:48:46,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:48:49,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:52,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:48:54,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:48:59,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:01,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:49:01,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:49:01,319 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:49:02,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:04,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:10,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:12,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:16,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:49:16,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:49:16,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:21,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:22,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:27,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:31,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:49:31,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:49:31,321 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:49:32,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:33,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:36,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:49:36,966 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:49:36,967 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:49:36,967 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:49:37,553 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:49:38,554 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:49:38,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:42,622 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:44,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:46,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:49:46,057 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:49:50,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:49:52,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:49:55,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:00,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:01,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:50:01,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:50:01,325 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:50:02,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:06,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:12,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:12,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:16,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:50:16,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:50:18,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:22,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:24,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:29,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:31,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:50:31,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:50:31,330 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:50:32,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:35,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:40,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:42,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:46,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:50:46,058 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:50:46,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:51,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:50:52,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:50:57,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:01,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:51:01,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:51:01,329 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:51:02,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:03,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:08,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:12,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:14,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:16,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:51:16,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:51:20,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:22,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:26,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:31,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:51:31,059 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:51:31,334 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:51:32,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:32,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:37,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:40,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:51:40,423 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:51:40,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:51:40,424 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:51:40,616 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:51:42,618 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:51:42,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:42,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:46,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:51:46,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:51:47,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:52,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:51:53,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:51:58,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:01,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:52:01,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:52:01,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:52:02,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:04,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:09,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:12,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:14,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:16,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:52:16,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:52:19,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:22,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:25,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:31,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:52:31,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:52:31,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:31,336 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:52:32,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:36,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:42,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:42,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:46,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:52:46,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:52:47,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:52,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:52:53,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:52:58,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:01,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:53:01,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:53:01,340 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:53:02,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:04,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:09,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:12,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:15,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:16,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:53:16,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:53:20,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:22,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:26,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:31,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:53:31,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:53:31,340 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:53:32,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:32,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:37,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:42,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:53:42,294 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:53:42,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:53:42,295 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:53:42,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:42,685 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:53:43,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:44,686 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:53:46,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:53:46,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:53:48,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:53:52,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:53:54,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:00,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:01,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:54:01,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:54:01,343 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:54:02,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:06,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:11,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:12,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:16,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:54:16,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:54:17,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:22,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:22,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:27,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:31,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:54:31,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:54:31,345 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:54:32,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:33,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:38,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:42,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:43,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:46,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:54:46,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:54:48,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:52,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:54:54,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:54:59,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:01,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:55:01,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:55:01,354 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:55:02,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:04,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:10,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:12,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:16,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:55:16,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:55:16,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:21,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:22,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:27,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:31,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:55:31,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:55:31,350 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:55:32,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:32,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:37,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:42,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:43,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:44,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:55:44,209 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:55:44,209 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:55:44,211 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:55:44,741 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:55:46,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:55:46,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:55:46,742 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:55:48,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:52,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:55:54,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:55:59,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:01,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:56:01,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:56:01,352 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:56:02,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:05,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:10,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:12,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:16,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:56:16,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:56:16,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:21,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:22,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:26,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:31,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:56:31,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:56:31,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:56:32,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:32,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:37,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:42,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:43,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:46,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:56:46,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:56:48,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:52,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:56:53,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:56:58,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:01,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:57:01,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:57:01,357 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:57:02,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:04,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:09,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:12,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:14,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:16,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:57:16,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:57:19,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:22,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:25,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:30,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:31,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:57:31,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:57:31,360 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:57:33,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:36,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:42,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:43,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:46,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:57:46,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:57:47,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:47,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:57:47,903 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:57:47,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:57:47,904 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:57:48,801 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:57:48,802 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:57:53,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:57:53,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:57:59,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:01,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:58:01,084 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:58:01,362 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:58:03,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:05,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:11,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:13,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:16,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:58:16,086 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:58:16,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:22,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:23,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:28,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:31,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:58:31,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:58:31,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:58:33,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:33,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:39,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:43,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:44,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:46,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:58:46,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:58:50,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:58:53,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:58:55,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:01,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:01,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:59:01,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:59:01,367 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:59:03,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:07,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:12,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:13,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:16,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:59:16,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:59:17,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:23,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:23,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:29,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:31,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:59:31,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:59:31,369 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 16:59:33,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:34,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:40,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:43,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:45,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:46,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 16:59:46,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 16:59:49,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 16:59:49,536 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 16:59:49,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 16:59:49,536 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 16:59:49,852 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 16:59:50,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 16:59:50,853 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 16:59:53,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 16:59:56,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:01,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:00:01,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:00:01,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:01,374 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:00:03,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:07,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:13,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:13,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:16,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:00:16,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:00:19,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:23,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:25,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:31,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:31,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:00:31,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:00:31,376 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:00:33,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:37,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:43,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:43,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:46,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:00:46,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:00:48,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:53,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:00:54,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:00:59,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:01,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:01:01,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:01:01,379 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:01:03,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:04,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:10,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:13,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:16,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:16,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:01:16,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:01:22,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:23,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:28,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:31,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:01:31,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:01:33,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:34,471 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:01:34,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:40,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:43,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:46,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:46,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:01:46,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:01:51,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:01:51,249 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:01:51,250 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:01:51,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:01:51,251 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:01:51,924 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:01:52,925 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:01:53,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:01:57,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:01,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:02:01,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:02:01,385 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:02:02,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:03,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:08,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:13,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:13,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:16,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:02:16,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:02:18,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:23,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:24,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:29,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:31,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:02:31,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:02:31,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:02:33,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:35,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:40,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:43,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:46,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:46,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:02:46,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:02:51,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:02:53,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:02:56,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:01,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:01,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:03:01,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:03:01,389 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:03:03,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:07,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:13,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:13,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:16,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:03:16,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:03:19,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:23,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:24,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:29,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:31,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:03:31,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:03:31,391 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:03:33,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:35,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:40,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:43,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:46,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:46,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:03:46,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:03:51,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:03:53,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:03:53,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:03:53,131 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:03:53,132 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:03:53,133 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:03:53,991 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:03:54,992 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:03:56,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:01,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:04:01,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:04:01,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:01,394 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:04:03,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:07,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:12,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:13,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:16,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:04:16,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:04:17,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:23,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:23,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:29,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:31,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:04:31,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:04:31,402 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:04:33,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:34,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:40,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:43,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:46,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:46,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:04:46,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:04:51,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:04:53,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:04:57,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:01,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:05:01,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:05:01,399 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:05:02,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:03,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:08,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:13,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:14,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:16,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:05:16,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:05:19,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:23,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:25,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:30,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:31,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:05:31,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:05:31,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:05:33,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:36,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:41,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:43,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:46,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:05:46,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:05:46,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:52,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:53,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:05:56,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:05:56,536 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:05:56,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:05:56,537 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:05:57,055 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:05:57,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:05:59,056 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:06:01,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:06:01,126 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:06:01,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:06:03,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:03,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:08,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:13,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:14,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:16,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:06:16,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:06:19,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:23,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:25,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:30,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:31,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:06:31,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:06:31,405 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:06:33,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:35,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:41,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:43,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:46,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:06:46,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:06:46,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:52,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:06:53,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:06:58,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:01,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:07:01,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:07:01,408 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:07:03,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:03,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:08,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:13,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:14,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:16,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:07:16,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:07:20,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:23,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:26,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:31,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:31,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:07:31,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:07:31,410 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:07:33,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:37,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:43,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:43,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:46,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:07:46,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:07:48,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:53,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:07:54,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:07:58,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:07:58,426 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:07:58,426 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:07:58,427 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:07:59,120 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:07:59,120 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:07:59,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:01,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:08:01,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:08:01,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:08:03,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:05,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:10,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:13,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:16,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:16,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:08:16,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:08:21,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:23,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:27,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:31,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:08:31,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:08:31,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:08:32,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:33,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:38,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:43,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:44,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:46,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:08:46,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:08:49,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:08:53,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:08:55,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:00,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:01,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:09:01,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:09:01,422 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:09:03,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:06,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:12,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:13,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:16,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:09:16,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:09:17,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:23,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:23,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:28,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:31,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:09:31,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:09:31,420 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:09:33,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:33,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:38,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:43,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:44,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:46,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:09:46,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:09:49,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:09:53,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:09:55,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:00,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:00,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:10:00,360 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:10:00,360 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:10:00,362 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:10:01,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:10:01,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:10:01,190 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:10:01,191 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:10:01,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:10:03,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:06,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:12,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:13,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:16,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:10:16,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:10:17,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:23,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:23,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:28,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:31,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:10:31,138 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:10:31,426 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:10:33,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:34,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:39,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:43,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:45,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:46,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:10:46,139 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:10:50,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:10:53,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:10:56,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:01,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:01,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:11:01,139 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:11:01,427 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:11:03,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:07,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:12,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:13,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:16,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:11:16,139 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:11:17,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:23,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:23,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:28,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:31,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:11:31,143 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:11:31,429 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:11:33,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:34,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:39,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:43,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:45,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:46,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:11:46,143 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:11:50,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:11:53,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:11:56,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:01,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:12:01,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:12:01,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:01,432 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:12:03,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:03,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:12:03,533 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:12:03,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:12:03,534 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:12:04,254 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:12:05,255 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:12:06,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:12,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:13,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:16,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:12:16,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:12:17,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:23,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:23,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:28,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:31,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:12:31,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:12:31,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:12:33,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:34,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:39,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:43,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:45,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:46,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:12:46,163 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:12:50,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:12:53,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:12:56,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:01,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:01,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:13:01,162 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:13:01,436 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:13:03,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:07,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:12,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:13,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:16,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:13:16,163 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:13:17,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:23,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:23,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:28,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:31,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:13:31,163 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:13:31,442 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:13:33,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:34,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:39,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:43,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:45,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:46,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:13:46,163 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:13:50,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:13:53,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:13:56,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:01,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:01,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:14:01,164 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:14:01,441 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:14:03,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:05,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:14:05,480 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:14:05,480 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:14:05,482 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:14:06,319 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:14:06,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:07,319 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:14:12,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:13,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:16,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:14:16,164 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:14:17,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:23,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:23,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:29,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:31,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:14:31,164 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:14:31,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:14:33,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:35,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:40,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:43,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:46,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:46,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:14:46,164 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:14:51,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:14:53,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:14:57,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:01,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:15:01,165 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:15:01,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:15:02,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:03,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:08,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:13,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:14,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:16,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:15:16,166 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:15:19,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:23,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:24,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:29,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:31,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:15:31,166 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:15:31,448 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:15:33,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:35,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:41,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:43,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:46,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:15:46,167 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:15:46,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:52,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:15:53,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:15:58,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:01,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:16:01,167 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:16:01,451 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:16:03,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:03,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:07,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:16:07,220 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:16:07,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:16:07,221 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:16:07,378 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:16:09,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:09,379 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:16:13,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:14,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:16,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:16:16,167 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:16:19,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:23,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:24,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:29,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:31,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:16:31,168 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:16:31,453 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:16:33,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:35,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:40,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:43,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:46,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:46,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:16:46,167 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:16:51,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:16:53,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:16:57,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:01,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:17:01,168 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:17:01,456 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:17:02,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:03,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:08,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:13,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:14,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:16,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:17:16,168 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:17:19,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:23,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:25,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:30,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:31,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:17:31,169 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:17:31,458 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:17:33,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:36,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:41,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:43,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:46,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:17:46,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:17:46,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:52,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:17:53,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:17:58,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:01,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:18:01,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:18:01,465 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:18:03,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:04,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:10,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:10,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:18:10,559 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:18:10,559 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:18:10,560 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:18:11,435 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:18:11,435 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:18:13,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:15,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:16,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:18:16,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:18:20,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:23,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:26,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:31,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:31,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:18:31,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:18:31,463 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:18:33,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:36,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:42,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:43,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:46,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:18:46,170 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:18:47,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:53,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:18:53,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:18:58,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:01,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:19:01,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:19:01,467 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:19:03,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:03,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:08,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:13,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:14,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:16,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:19:16,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:19:19,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:23,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:25,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:30,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:31,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:19:31,172 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:19:31,469 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:19:33,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:36,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:41,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:43,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:46,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:19:46,172 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:19:46,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:52,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:19:53,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:19:58,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:01,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:20:01,172 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:20:01,471 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:20:03,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:03,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:09,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:12,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:20:12,174 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:20:12,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:20:12,175 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:20:12,489 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:20:13,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:13,490 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:20:15,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:16,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:20:16,172 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:20:21,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:23,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:27,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:31,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:20:31,172 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:20:31,473 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:20:32,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:33,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:38,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:43,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:44,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:46,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:20:46,173 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:20:49,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:20:53,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:20:55,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:00,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:01,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:21:01,173 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:21:01,476 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:21:03,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:06,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:11,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:13,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:16,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:21:16,173 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:21:16,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:22,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:23,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:28,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:31,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:21:31,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:21:31,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:21:33,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:34,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:40,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:43,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:46,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:46,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:21:46,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:21:52,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:21:53,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:21:57,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:01,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:22:01,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:22:01,480 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:22:02,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:03,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:08,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:13,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:13,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:13,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:22:13,880 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:22:13,880 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:22:13,881 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:22:14,548 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:22:15,549 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:22:16,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:22:16,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:22:18,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:23,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:24,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:29,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:31,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:22:31,174 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:22:31,483 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:22:33,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:35,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:40,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:43,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:46,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:46,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:22:46,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:22:51,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:22:53,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:22:57,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:01,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:23:01,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:23:01,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:23:02,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:03,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:08,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:13,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:13,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:16,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:23:16,175 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:23:18,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:23,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:24,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:29,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:31,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:23:31,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:23:31,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:23:34,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:34,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:40,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:44,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:45,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:46,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:23:46,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:23:51,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:23:54,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:23:56,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:01,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:24:01,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:24:01,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:01,492 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:24:04,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:07,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:12,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:14,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:16,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:24:16,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:24:17,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:24:17,281 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:24:17,281 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:24:17,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:17,282 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:24:17,599 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:24:19,600 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:24:23,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:24,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:28,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:31,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:24:31,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:24:31,493 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:24:33,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:34,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:39,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:44,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:45,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:46,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:24:46,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:24:50,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:24:54,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:24:55,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:01,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:25:01,177 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:25:01,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:01,495 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:25:04,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:07,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:13,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:14,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:16,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:25:16,176 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:25:18,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:24,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:24,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:29,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:31,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:25:31,177 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:25:31,498 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:25:34,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:34,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:40,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:44,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:46,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:25:46,177 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:25:46,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:52,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:25:54,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:25:58,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:01,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:26:01,177 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:26:01,500 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:26:03,501 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:04,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:09,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:14,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:15,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:16,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:26:16,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:26:19,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:26:19,128 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:26:19,128 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:26:19,129 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:26:19,648 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:26:21,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:21,650 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:26:24,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:26,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:31,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:26:31,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:26:31,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:31,503 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:26:34,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:37,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:43,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:44,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:46,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:26:46,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:26:48,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:54,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:26:54,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:26:59,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:01,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:27:01,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:27:01,506 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:27:04,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:04,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:10,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:14,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:15,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:16,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:27:16,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:27:20,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:24,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:26,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:31,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:27:31,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:27:31,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:31,512 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:27:34,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:37,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:42,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:44,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:46,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:27:46,191 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:27:47,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:53,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:27:54,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:27:59,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:01,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:28:01,180 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:28:01,510 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:28:04,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:05,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:10,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:14,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:16,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:28:16,182 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:28:16,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:20,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:28:20,909 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:28:20,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:28:20,911 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:28:21,700 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:28:21,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:23,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:28:24,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:27,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:31,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:28:31,182 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:28:31,512 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:28:32,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:34,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:38,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:43,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:44,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:46,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:28:46,183 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:28:48,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:54,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:28:54,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:28:59,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:01,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:29:01,183 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:29:01,516 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:29:04,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:05,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:11,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:14,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:16,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:29:16,183 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:29:16,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:22,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:24,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:28,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:31,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:29:31,183 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:29:31,517 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:29:33,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:34,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:39,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:44,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:45,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:46,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:29:46,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:29:50,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:29:54,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:29:56,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:01,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:30:01,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:30:01,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:01,520 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:30:04,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:07,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:13,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:14,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:16,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:30:16,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:30:18,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:24,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:30:24,239 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:30:24,240 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:30:24,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:24,241 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:30:24,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:24,753 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:30:25,754 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:30:29,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:31,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:30:31,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:30:31,522 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:30:34,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:34,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:40,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:44,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:45,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:46,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:30:46,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:30:51,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:30:54,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:30:57,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:01,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:31:01,184 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:31:01,524 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:31:02,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:04,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:08,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:13,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:14,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:16,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:31:16,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:31:18,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:24,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:24,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:29,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:31,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:31:31,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:31:31,526 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:31:34,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:35,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:40,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:44,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:46,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:31:46,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:31:46,319 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:51,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:31:54,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:31:56,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:01,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:32:01,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:32:01,532 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:32:01,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:04,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:07,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:13,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:14,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:16,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:32:16,185 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:32:19,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:24,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:24,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:26,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:32:26,170 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:32:26,171 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:32:26,171 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:32:26,807 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:32:27,808 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:32:29,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:31,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:32:31,186 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:32:31,531 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:32:34,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:34,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:39,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:44,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:45,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:46,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:32:46,186 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:32:51,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:32:54,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:32:57,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:01,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:33:01,186 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:33:01,535 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:33:02,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:04,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:08,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:13,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:14,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:16,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:33:16,186 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:33:18,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:24,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:24,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:29,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:31,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:33:31,187 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:33:31,536 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:33:34,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:35,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:41,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:44,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:46,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:33:46,187 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:33:46,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:52,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:33:54,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:33:58,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:01,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:34:01,188 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:34:01,538 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:34:03,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:04,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:09,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:14,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:15,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:16,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:34:16,187 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:34:20,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:24,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:25,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:28,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:34:28,679 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:34:28,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:34:28,681 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:34:28,861 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:34:29,863 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:34:31,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:34:31,188 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:34:31,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:31,541 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:34:34,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:36,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:41,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:44,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:46,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:34:46,189 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:34:46,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:52,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:34:54,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:34:58,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:01,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:35:01,202 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:35:01,543 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:35:03,545 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:04,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:09,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:14,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:15,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:16,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:35:16,201 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:35:20,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:24,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:25,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:30,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:31,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:35:31,202 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:35:31,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:35:34,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:36,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:41,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:44,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:46,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:35:46,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:35:46,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:52,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:35:54,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:35:58,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:01,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:36:01,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:36:01,549 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:36:03,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:04,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:09,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:14,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:15,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:16,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:36:16,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:36:20,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:24,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:26,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:31,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:36:31,214 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:36:31,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:31,555 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:36:32,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:36:32,094 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:36:32,094 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:36:32,094 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:36:32,924 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:36:33,925 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:36:34,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:37,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:43,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:44,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:46,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:36:46,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:36:48,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:54,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:36:54,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:36:59,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:01,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:37:01,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:37:01,554 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:37:04,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:05,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:10,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:14,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:15,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:16,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:37:16,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:37:21,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:24,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:27,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:31,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:37:31,215 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:37:31,558 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:37:32,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:34,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:38,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:44,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:44,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:46,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:37:46,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:37:50,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:37:54,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:37:56,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:01,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:38:01,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:38:01,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:01,559 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:38:04,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:07,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:13,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:14,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:16,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:38:16,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:38:18,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:24,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:24,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:29,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:31,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:38:31,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:38:31,562 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:38:33,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:38:33,748 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:38:33,748 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:38:33,749 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:38:34,000 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:38:34,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:35,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:36,001 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:38:41,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:44,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:46,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:38:46,216 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:38:46,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:52,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:38:54,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:38:58,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:01,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:39:01,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:39:01,565 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:39:03,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:04,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:09,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:14,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:15,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:16,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:39:16,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:39:20,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:24,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:26,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:31,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:39:31,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:39:31,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:31,567 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:39:34,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:37,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:43,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:44,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:46,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:39:46,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:39:48,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:54,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:39:54,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:39:59,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:01,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:40:01,217 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:40:01,572 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:40:04,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:04,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:09,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:14,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:15,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:16,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:40:16,218 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:40:21,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:24,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:27,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:31,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:40:31,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:40:31,572 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:40:32,573 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:34,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:35,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:40:35,416 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:40:35,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:40:35,419 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:40:36,067 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:40:38,067 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:40:38,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:44,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:44,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:46,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:40:46,218 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:40:49,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:54,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:40:54,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:40:59,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:01,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:41:01,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:41:01,579 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:41:04,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:05,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:10,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:14,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:15,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:16,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:41:16,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:41:21,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:24,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:26,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:31,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:41:31,219 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:41:31,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:31,577 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:41:34,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:37,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:43,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:44,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:46,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:41:46,220 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:41:48,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:54,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:41:54,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:41:59,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:01,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:42:01,220 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:42:01,580 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:42:04,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:04,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:09,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:14,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:15,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:16,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:42:16,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:42:20,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:24,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:26,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:31,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:42:31,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:42:31,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:31,582 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:42:34,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:36,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:42:36,855 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:42:36,855 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:42:36,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:36,856 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:42:37,132 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:42:38,132 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:42:42,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:44,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:46,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:42:46,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:42:47,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:53,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:42:54,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:42:59,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:01,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:43:01,221 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:43:01,585 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:43:04,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:05,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:10,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:14,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:15,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:16,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:43:16,222 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:43:21,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:24,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:27,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:31,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:43:31,222 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:43:31,587 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:43:32,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:34,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:38,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:43,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:44,300 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:46,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:43:46,222 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:43:49,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:54,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:43:54,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:43:59,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:01,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:44:01,223 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:44:01,590 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:44:04,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:04,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:10,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:14,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:16,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:44:16,223 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:44:16,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:22,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:24,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:28,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:31,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:44:31,224 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:44:31,593 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:44:33,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:34,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:39,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:40,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:44:40,127 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:44:40,127 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:44:40,128 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:44:40,201 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:44:42,203 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:44:44,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:44,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:46,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:44:46,224 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:44:49,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:44:54,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:44:55,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:00,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:01,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:45:01,224 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:45:01,596 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:45:04,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:06,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:12,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:14,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:16,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:45:16,225 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:45:17,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:23,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:24,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:29,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:31,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:45:31,224 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:45:31,598 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:45:34,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:34,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:39,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:44,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:45,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:46,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:45:46,224 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:45:50,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:45:54,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:45:56,383 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:01,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:46:01,225 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:46:04,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:05,158 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:46:05,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:10,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:14,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:15,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:16,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:46:16,225 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:46:21,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:24,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:27,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:31,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:46:31,226 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:46:31,603 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:46:32,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:34,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:38,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:42,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:46:42,053 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:46:42,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:46:42,056 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:46:42,268 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:46:44,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:44,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:46:44,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:46,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:46:46,225 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:46:49,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:46:54,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:46:55,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:00,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:01,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:47:01,226 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:47:01,606 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:47:04,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:06,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:11,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:14,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:16,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:47:16,227 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:47:16,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:22,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:24,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:28,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:31,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:47:31,227 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:47:31,611 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:47:33,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:34,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:39,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:44,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:45,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:46,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:47:46,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:47:50,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:47:54,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:47:56,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:01,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:48:01,227 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:48:01,612 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:48:01,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:04,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:07,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:12,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:14,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:16,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:48:16,227 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:48:17,441 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:23,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:24,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:29,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:31,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:48:31,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:48:31,614 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:48:34,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:35,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:40,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:43,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:48:43,833 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:48:43,833 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:48:43,834 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:48:44,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:44,339 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:48:45,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:46,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:48:46,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:48:46,340 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:48:50,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:48:54,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:48:55,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:00,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:01,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:49:01,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:49:01,616 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:49:04,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:06,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:11,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:14,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:16,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:49:16,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:49:16,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:22,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:24,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:27,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:31,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:49:31,229 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:49:31,619 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:49:32,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:34,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:38,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:44,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:44,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:46,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:49:46,229 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:49:49,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:49:54,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:49:55,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:00,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:01,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:50:01,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:50:01,621 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:50:04,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:06,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:12,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:14,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:16,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:50:16,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:50:18,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:23,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:24,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:29,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:31,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:50:31,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:50:31,627 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:50:34,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:34,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:40,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:44,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:45,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:46,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:50:46,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:50:47,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:50:47,290 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:50:47,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:50:47,291 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:50:47,424 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:50:48,425 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:50:50,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:50:54,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:50:56,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:01,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:51:01,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:51:01,435 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:01,626 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:51:04,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:07,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:12,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:14,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:16,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:51:16,231 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:51:18,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:23,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:24,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:29,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:31,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:51:31,231 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:51:31,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:51:34,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:35,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:41,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:44,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:46,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:51:46,231 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:51:46,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:52,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:51:54,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:51:57,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:01,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:52:01,231 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:52:01,631 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:52:02,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:04,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:08,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:14,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:14,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:16,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:52:16,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:52:19,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:24,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:25,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:30,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:31,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:52:31,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:52:31,634 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:52:34,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:35,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:41,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:44,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:46,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:52:46,233 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:52:46,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:48,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:52:48,994 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:52:48,994 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:52:48,996 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:52:49,501 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:52:50,501 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:52:52,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:52:54,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:52:58,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:01,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:53:01,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:53:01,636 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:53:03,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:04,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:09,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:14,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:15,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:16,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:53:16,232 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:53:20,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:24,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:26,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:31,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:53:31,233 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:53:31,639 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:53:31,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:34,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:37,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:43,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:44,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:46,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:53:46,234 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:53:49,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:54,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:53:54,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:53:59,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:01,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:54:01,234 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:54:01,642 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:54:04,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:04,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:10,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:14,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:16,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:54:16,234 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:54:16,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:22,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:24,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:28,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:31,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:54:31,234 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:54:31,644 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:54:33,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:34,345 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:39,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:44,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:45,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:46,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:54:46,235 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:54:50,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:54:50,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:54:50,725 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:54:50,726 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:54:50,726 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:54:51,578 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:54:52,579 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:54:54,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:54:56,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:01,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:55:01,236 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:55:01,650 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:55:01,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:04,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:07,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:13,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:14,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:16,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:55:16,235 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:55:19,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:24,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:24,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:29,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:31,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:55:31,235 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:55:31,649 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:55:34,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:35,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:40,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:44,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:46,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:55:46,236 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:55:46,460 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:52,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:55:54,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:55:58,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:01,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:56:01,238 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:56:01,654 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:56:03,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:04,347 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:09,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:14,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:15,428 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:16,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:56:16,239 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:56:20,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:24,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:26,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:31,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:56:31,239 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:56:31,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:56:31,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:34,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:37,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:42,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:44,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:46,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:56:46,239 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:56:48,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:53,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:56:54,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:56:54,018 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:56:54,019 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:56:54,019 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:56:54,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:56:54,655 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:56:55,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:56:59,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:01,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:57:01,243 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:57:01,657 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:57:04,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:04,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:10,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:14,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:16,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:57:16,240 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:57:16,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:21,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:24,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:27,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:31,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:57:31,247 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:57:31,659 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:57:32,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:34,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:38,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:44,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:44,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:46,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:57:46,247 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:57:49,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:57:54,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:57:55,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:00,439 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:01,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:58:01,248 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:58:01,673 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:58:04,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:05,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:11,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:14,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:16,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:58:16,248 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:58:16,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:22,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:24,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:28,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:31,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:58:31,248 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:58:34,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:34,951 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:58:34,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:40,442 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:44,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:46,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:58:46,248 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:58:46,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:52,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:58:54,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:58:55,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 17:58:55,943 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 17:58:55,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 17:58:55,944 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 17:58:56,718 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 17:58:57,719 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 17:58:57,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:01,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:59:01,249 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:59:01,680 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:59:03,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:04,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:09,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:14,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:15,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:16,248 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:59:16,249 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:59:20,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:24,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:26,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:31,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:59:31,249 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:59:31,688 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 17:59:31,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:34,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:37,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:42,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:44,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:46,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 17:59:46,249 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 17:59:48,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:53,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 17:59:54,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 17:59:59,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:01,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:00:01,250 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:00:01,685 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:00:04,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:05,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:11,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:14,369 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:16,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:00:16,268 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:00:16,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:22,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:24,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:28,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:31,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:00:31,269 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:00:31,688 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:00:33,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:34,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:39,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:44,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:45,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:46,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:00:46,269 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:00:51,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:54,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:00:57,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:00:57,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:00:57,480 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:00:57,481 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:00:57,482 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:00:57,786 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:00:59,788 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:01:01,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:01:01,269 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:01:04,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:04,592 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:01:04,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:10,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:14,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:15,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:16,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:01:16,270 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:01:21,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:24,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:27,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:31,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:01:31,270 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:01:31,692 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:01:32,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:34,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:38,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:43,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:44,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:46,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:01:46,270 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:01:48,456 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:54,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:01:54,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:01:59,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:01,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:02:01,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:02:01,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:02:04,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:05,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:10,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:14,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:16,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:02:16,271 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:02:16,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:21,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:24,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:27,452 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:31,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:02:31,270 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:02:31,697 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:02:32,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:34,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:38,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:44,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:44,453 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:46,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:02:46,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:02:50,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:02:54,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:02:56,454 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:00,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:03:00,737 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:03:00,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:03:00,738 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:03:00,851 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:03:01,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:03:01,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:03:01,502 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:01,701 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:03:01,852 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:03:04,373 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:07,462 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:12,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:14,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:16,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:03:16,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:03:18,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:23,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:24,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:29,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:31,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:03:31,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:03:31,703 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:03:34,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:34,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:40,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:44,374 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:46,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:03:46,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:03:46,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:52,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:03:54,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:03:57,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:01,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:04:01,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:04:01,705 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:04:02,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:04,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:08,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:14,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:14,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:16,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:04:16,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:04:19,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:24,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:25,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:30,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:31,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:04:31,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:04:31,712 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:04:34,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:35,467 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:40,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:44,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:46,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:04:46,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:04:46,464 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:51,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:04:54,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:04:57,468 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:01,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:05:01,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:05:01,711 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:05:02,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:05:02,588 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:05:02,588 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:05:02,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:02,591 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:05:02,915 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:05:03,917 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:05:04,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:08,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:14,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:14,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:16,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:05:16,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:05:20,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:24,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:25,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:31,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:05:31,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:05:31,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:31,713 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:05:34,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:36,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:42,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:44,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:46,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:05:46,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:05:47,508 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:53,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:05:54,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:05:59,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:01,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:06:01,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:06:01,717 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:06:04,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:04,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:09,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:14,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:15,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:06:16,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:06:20,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:24,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:26,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:31,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:06:31,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:06:31,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:31,718 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:06:34,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:37,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:43,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:44,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:46,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:06:46,291 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:06:48,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:54,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:06:54,473 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:06:59,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:01,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:07:01,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:07:01,720 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:07:04,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:04,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:07:04,440 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:07:04,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:07:04,440 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:07:04,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:04,980 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:07:05,981 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:07:10,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:14,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:15,474 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:16,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:07:16,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:07:20,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:24,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:26,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:31,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:07:31,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:07:31,723 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:07:31,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:34,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:37,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:43,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:44,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:46,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:07:46,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:07:49,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:54,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:07:54,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:07:59,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:01,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:08:01,292 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:08:01,725 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:08:04,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:05,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:10,476 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:14,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:16,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:08:16,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:08:16,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:21,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:24,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:27,477 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:31,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:08:31,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:08:31,728 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:08:32,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:34,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:38,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:44,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:44,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:46,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:08:46,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:08:49,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:08:54,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:08:55,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:00,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:01,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:09:01,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:09:01,733 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:09:04,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:05,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:07,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:09:07,903 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:09:07,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:09:07,904 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:09:08,052 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:09:10,053 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:09:10,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:14,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:16,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:09:16,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:09:16,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:21,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:24,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:27,488 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:31,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:09:31,293 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:09:31,732 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:09:32,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:34,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:38,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:44,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:44,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:46,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:09:46,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:09:49,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:09:54,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:09:55,489 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:00,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:01,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:10:01,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:10:01,736 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:10:04,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:06,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:11,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:14,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:16,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:10:16,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:10:17,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:22,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:24,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:28,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:31,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:10:31,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:10:31,737 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:10:33,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:34,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:39,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:44,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:45,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:46,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:10:46,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:10:50,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:10:54,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:10:55,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:01,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:11:01,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:11:01,506 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:01,739 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:11:04,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:07,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:09,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:11:09,651 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:11:09,652 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:11:09,652 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:11:10,110 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:11:10,111 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:11:12,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:14,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:16,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:11:16,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:11:18,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:24,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:24,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:29,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:31,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:11:31,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:11:31,741 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:11:34,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:35,493 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:40,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:44,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:46,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:11:46,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:11:46,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:51,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:11:54,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:11:57,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:01,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:12:01,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:12:01,745 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:12:02,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:04,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:08,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:14,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:14,495 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:12:16,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:12:19,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:24,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:25,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:31,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:12:31,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:12:31,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:31,748 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:12:34,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:36,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:41,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:44,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:46,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:12:46,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:12:47,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:52,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:12:54,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:12:57,498 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:01,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:13:01,297 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:13:01,751 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:13:02,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:04,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:08,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:11,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:13:11,769 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:13:11,769 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:13:11,770 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:13:12,184 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:13:13,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:14,185 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:13:14,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:16,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:13:16,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:13:19,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:24,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:25,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:30,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:31,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:13:31,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:13:31,756 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:13:34,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:35,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:40,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:44,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:46,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:13:46,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:13:46,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:51,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:13:54,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:13:57,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:01,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:14:01,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:14:01,756 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:14:02,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:04,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:08,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:14,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:14,515 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:16,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:14:16,298 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:14:19,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:24,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:24,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:29,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:31,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:14:31,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:14:31,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:14:34,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:35,517 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:40,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:44,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:46,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:14:46,309 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:14:46,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:52,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:14:54,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:14:58,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:01,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:15:01,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:15:01,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:15:03,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:04,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:09,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:13,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:15:13,821 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:15:13,821 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:15:13,822 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:15:14,266 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:15:14,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:15,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:16,267 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:15:16,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:15:16,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:15:20,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:25,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:25,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:30,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:31,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:15:31,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:15:31,763 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:15:35,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:35,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:41,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:45,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:46,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:15:46,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:15:47,480 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:52,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:15:55,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:15:58,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:01,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:16:01,310 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:16:01,765 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:16:04,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:05,935 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:09,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:14,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:15,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:16,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:16:16,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:16:20,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:25,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:25,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:30,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:31,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:16:31,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:16:31,768 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:16:35,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:35,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:41,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:45,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:46,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:16:46,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:16:47,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:52,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:16:55,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:16:58,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:01,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:17:01,327 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:17:01,770 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:17:04,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:05,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:09,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:14,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:15,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:16,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:17:16,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:17:17,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:17:17,099 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:17:17,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:17:17,101 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:17:17,327 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:17:18,327 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:17:20,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:25,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:25,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:30,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:31,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:17:31,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:17:31,774 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:17:35,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:36,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:41,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:45,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:46,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:17:46,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:17:47,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:52,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:17:55,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:17:58,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:01,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:18:01,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:18:01,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:18:04,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:05,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:09,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:14,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:15,938 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:16,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:18:16,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:18:20,510 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:25,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:25,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:30,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:31,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:18:31,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:18:31,779 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:18:35,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:36,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:41,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:45,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:46,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:18:46,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:18:47,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:52,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:18:55,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:18:57,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:01,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:19:01,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:19:01,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:19:03,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:05,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:08,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:13,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:15,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:16,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:19:16,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:19:18,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:19:18,883 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:19:18,883 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:19:18,884 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:19:19,384 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:19:19,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:20,385 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:19:24,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:25,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:30,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:31,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:19:31,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:19:31,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:19:35,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:36,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:41,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:45,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:46,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:19:46,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:19:47,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:52,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:19:55,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:19:57,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:01,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:20:01,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:20:01,787 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:20:03,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:05,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:08,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:13,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:15,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:16,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:20:16,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:20:19,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:24,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:25,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:30,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:31,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:20:31,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:20:31,789 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:20:35,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:36,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:41,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:45,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:46,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:20:46,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:20:47,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:52,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:20:55,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:20:58,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:01,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:21:01,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:21:05,402 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:21:05,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:05,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:10,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:15,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:16,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:21:16,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:21:16,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:20,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:21:20,568 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:21:20,568 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:21:20,569 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:21:21,444 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:21:21,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:22,445 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:21:25,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:27,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:31,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:21:31,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:21:31,857 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:21:33,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:35,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:38,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:44,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:45,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:46,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:21:46,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:21:50,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:21:55,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:21:55,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:00,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:01,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:22:01,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:22:01,861 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:22:05,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:06,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:11,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:15,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:16,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:22:16,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:22:17,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:22,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:25,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:28,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:31,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:22:31,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:22:31,876 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:22:34,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:35,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:39,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:45,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:45,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:46,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:22:46,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:22:50,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:22:55,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:22:56,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:01,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:23:01,335 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:23:01,864 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:23:02,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:05,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:07,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:13,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:15,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:16,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:23:16,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:23:19,486 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:24,550 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:23:24,552 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:23:24,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:23:24,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:24,556 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:23:25,505 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:23:25,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:26,506 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:23:29,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:31,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:23:31,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:23:31,867 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:23:35,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:35,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:40,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:45,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:46,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:23:46,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:23:46,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:51,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:23:55,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:23:56,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:01,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:24:01,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:24:01,876 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:24:02,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:05,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:07,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:12,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:15,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:16,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:24:16,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:24:18,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:23,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:25,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:29,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:31,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:24:31,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:24:31,872 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:24:35,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:35,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:40,972 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:45,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:46,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:24:46,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:24:46,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:51,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:24:55,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:24:57,973 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:01,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:25:01,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:25:01,875 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:25:03,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:05,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:08,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:13,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:15,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:16,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:25:16,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:25:19,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:24,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:25,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:26,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:25:26,463 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:25:26,463 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:25:26,464 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:25:26,565 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:25:28,566 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:25:30,503 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:31,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:25:31,337 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:25:31,877 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:25:35,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:35,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:40,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:45,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:46,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:25:46,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:25:46,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:51,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:25:55,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:25:57,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:01,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:26:01,338 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:26:01,880 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:26:03,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:05,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:08,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:13,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:15,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:16,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:26:16,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:26:19,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:24,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:25,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:29,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:31,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:26:31,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:26:31,882 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:26:35,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:35,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:41,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:45,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:46,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:26:46,339 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:26:46,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:52,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:26:55,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:26:58,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:01,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:27:01,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:27:01,884 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:27:03,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:05,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:09,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:14,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:15,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:16,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:27:16,340 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:27:19,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:25,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:25,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:28,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:27:28,285 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:27:28,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:27:28,286 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:27:28,628 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:27:30,320 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:30,630 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:27:31,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:27:31,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:27:31,887 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:27:35,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:35,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:41,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:45,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:46,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:27:46,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:27:46,509 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:52,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:27:55,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:27:57,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:01,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:28:01,341 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:28:01,889 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:28:02,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:05,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:08,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:14,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:15,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:16,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:28:16,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:28:19,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:25,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:25,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:31,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:31,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:28:31,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:28:31,894 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:28:35,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:37,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:42,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:45,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:46,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:28:46,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:28:47,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:53,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:28:55,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:28:59,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:01,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:29:01,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:29:01,897 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:29:04,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:05,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:10,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:15,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:15,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:16,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:29:16,342 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:29:20,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:25,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:26,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:31,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:31,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:29:31,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:29:31,899 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:29:32,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:29:32,923 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:29:32,924 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:29:32,925 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:29:33,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:29:34,702 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:29:35,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:37,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:42,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:45,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:46,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:29:46,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:29:47,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:53,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:29:55,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:29:59,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:01,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:30:01,343 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:30:01,902 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:30:04,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:05,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:10,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:15,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:15,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:16,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:30:16,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:30:20,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:25,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:26,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:31,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:31,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:30:31,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:30:31,905 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:30:35,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:37,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:42,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:45,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:46,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:30:46,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:30:47,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:53,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:30:55,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:30:58,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:01,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:31:01,344 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:31:01,907 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:31:03,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:05,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:09,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:15,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:15,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:16,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:31:16,394 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:31:20,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:25,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:26,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:31,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:31,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:31:31,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:31:31,910 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:31:35,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:31:35,151 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:31:35,151 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:31:35,152 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:31:35,770 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:31:35,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:36,771 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:31:37,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:43,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:45,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:46,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:31:46,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:31:48,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:54,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:31:55,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:31:59,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:01,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:32:01,400 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:32:05,035 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:32:05,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:05,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:11,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:15,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:16,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:16,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:32:16,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:32:21,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:25,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:26,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:31,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:31,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:32:31,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:32:31,915 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:32:35,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:37,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:42,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:45,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:46,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:32:46,401 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:32:47,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:53,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:32:55,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:32:59,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:01,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:33:01,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:33:01,919 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:33:04,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:05,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:10,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:15,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:15,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:16,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:33:16,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:33:20,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:25,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:26,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:31,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:31,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:33:31,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:33:31,920 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:33:35,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:36,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:37,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:33:37,367 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:33:37,368 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:33:37,368 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:33:37,842 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:33:38,843 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:33:41,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:45,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:46,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:33:46,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:33:46,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:52,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:33:55,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:33:57,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:01,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:34:01,403 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:34:01,923 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:34:02,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:05,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:08,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:14,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:15,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:16,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:34:16,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:34:19,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:25,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:25,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:30,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:31,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:34:31,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:34:31,925 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:34:35,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:35,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:41,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:45,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:46,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:34:46,404 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:34:46,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:52,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:34:55,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:34:58,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:01,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:35:01,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:35:01,927 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:35:03,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:05,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:35:09,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:15,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:15,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:35:16,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:35:16,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:35:20,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:25,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:35:26,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:31,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:31,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:35:31,405 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:35:31,930 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:35:35,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:35:37,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:40,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:35:40,917 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:35:40,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:35:40,919 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:35:41,914 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:35:42,915 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:35:43,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:46,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:35:46,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:35:46,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:35:48,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:54,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:35:56,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:00,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:01,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:36:01,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:36:01,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:36:05,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:06,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:11,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:16,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:16,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:36:16,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:36:16,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:22,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:26,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:28,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:31,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:36:31,406 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:36:31,939 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:36:33,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:36,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:39,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:44,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:46,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:36:46,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:36:46,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:36:49,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:55,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:36:56,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:01,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:01,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:37:01,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:37:01,938 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:37:06,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:06,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:11,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:16,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:16,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:37:16,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:37:16,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:22,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:26,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:27,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:31,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:37:31,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:37:32,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:37:32,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:36,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:38,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:43,466 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:37:43,468 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:37:43,469 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:37:43,469 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:43,470 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:37:43,985 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:37:44,986 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:37:46,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:46,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:37:46,408 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:37:48,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:54,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:37:56,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:37:59,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:01,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:38:01,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:38:01,943 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:38:04,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:06,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:10,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:15,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:16,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:16,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:38:16,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:38:20,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:26,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:26,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:31,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:31,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:38:31,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:38:31,947 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:38:36,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:36,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:42,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:46,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:46,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:38:46,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:38:47,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:53,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:38:56,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:38:59,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:01,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:39:01,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:39:01,949 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:39:04,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:06,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:10,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:15,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:16,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:16,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:39:16,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:39:20,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:26,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:26,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:31,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:39:31,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:39:31,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:31,951 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:39:36,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:37,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:43,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:45,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:39:45,639 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:39:45,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:39:45,640 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:39:46,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:46,070 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:39:46,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:39:46,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:39:47,070 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:39:48,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:54,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:39:56,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:39:59,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:01,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:40:01,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:40:01,953 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:40:04,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:06,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:40:10,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:16,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:40:16,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:16,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:40:16,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:40:21,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:26,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:40:27,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:31,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:40:31,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:40:31,955 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:40:32,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:36,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:40:38,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:43,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:46,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:40:46,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:40:46,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:40:48,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:54,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:40:56,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:00,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:01,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:41:01,417 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:41:01,965 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:41:05,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:06,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:11,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:16,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:16,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:16,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:41:16,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:41:22,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:27,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:27,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:31,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:41:31,418 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:41:31,960 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:41:32,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:37,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:37,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:42,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:46,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:41:46,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:41:47,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:48,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:49,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:41:49,407 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:41:49,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:41:49,409 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:41:50,149 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:41:51,150 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:41:53,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:41:57,561 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:41:58,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:01,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:42:01,419 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:42:01,963 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:42:04,565 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:07,560 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:42:10,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:16,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:42:16,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:42:16,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:17,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:42:22,574 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:27,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:27,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:42:31,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:42:31,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:42:31,967 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:42:33,575 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:37,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:42:38,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:44,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:46,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:42:46,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:42:47,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:42:50,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:55,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:42:57,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:00,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:01,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:43:01,420 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:43:01,968 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:43:06,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:07,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:11,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:16,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:43:16,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:43:16,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:17,577 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:22,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:27,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:28,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:31,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:43:31,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:43:31,971 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:43:34,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:37,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:40,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:45,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:46,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:43:46,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:43:47,578 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:43:51,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:43:51,105 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:43:51,106 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:43:51,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:51,107 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:43:51,214 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:43:53,215 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:43:56,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:43:57,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:01,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:44:01,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:44:01,972 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:44:01,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:07,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:07,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:12,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:16,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:44:16,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:44:17,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:17,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:22,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:27,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:28,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:31,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:44:31,421 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:44:31,975 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:44:33,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:37,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:39,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:44,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:46,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:44:46,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:44:47,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:44:50,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:55,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:44:57,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:00,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:01,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:45:01,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:45:01,978 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:45:06,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:07,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:12,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:16,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:45:16,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:45:17,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:18,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:23,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:27,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:29,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:31,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:45:31,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:45:31,983 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:45:35,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:37,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:40,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:45,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:46,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:45:46,422 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:45:47,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:45:51,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:52,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:45:52,945 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:45:52,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:45:52,946 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:45:53,268 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:45:55,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:45:56,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:45:57,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:01,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:46:01,423 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:46:01,982 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:46:02,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:07,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:08,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:13,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:16,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:46:16,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:46:17,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:19,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:24,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:27,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:30,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:31,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:46:31,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:46:31,986 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:46:35,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:37,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:41,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:46,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:46:46,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:46:46,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:47,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:52,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:46:57,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:46:58,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:01,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:47:01,424 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:47:01,987 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:47:03,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:07,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:09,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:14,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:16,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:47:16,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:47:17,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:20,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:25,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:27,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:31,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:47:31,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:47:31,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:31,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:47:36,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:37,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:42,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:46,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:47:46,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:47:47,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:48,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:53,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:47:56,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:47:56,100 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:47:56,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:47:56,102 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:47:56,317 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:47:57,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:47:59,318 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:47:59,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:01,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:48:01,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:48:01,992 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:48:04,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:07,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:10,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:15,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:16,425 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:48:16,425 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:48:17,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:21,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:26,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:27,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:31,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:48:31,443 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:48:31,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:31,994 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:48:37,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:37,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:42,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:46,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:48:46,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:48:47,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:48,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:53,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:48:57,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:48:59,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:01,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:49:01,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:49:01,997 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:49:04,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:07,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:09,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:15,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:16,443 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:49:16,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:49:17,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:21,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:26,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:27,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:31,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:49:31,444 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:49:31,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:31,999 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:49:37,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:37,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:43,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:46,444 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:49:46,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:49:47,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:49,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:54,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:49:57,598 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:49:57,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:49:57,803 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:49:57,803 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:49:57,804 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:49:58,375 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:49:59,376 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:49:59,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:01,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:50:01,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:50:02,007 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:50:05,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:07,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:50:11,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:16,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:50:16,446 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:50:16,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:17,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:50:22,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:27,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:50:28,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:31,445 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:50:31,445 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:50:32,004 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:50:33,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:37,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:50:39,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:44,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:46,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:50:46,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:50:47,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:50:49,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:55,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:50:57,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:01,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:51:01,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:51:01,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:02,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:51:07,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:07,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:12,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:16,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:51:16,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:51:17,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:18,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:23,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:27,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:29,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:31,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:51:31,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:51:32,011 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:51:34,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:37,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:39,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:44,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:46,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:51:46,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:51:47,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:50,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:55,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:51:57,602 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:51:59,437 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:51:59,439 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:51:59,440 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:51:59,441 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:52:00,433 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:52:01,433 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:52:01,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:52:01,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:52:01,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:02,014 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:52:06,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:07,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:12,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:16,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:52:16,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:52:17,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:18,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:23,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:27,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:28,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:31,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:52:31,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:52:32,016 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:52:34,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:37,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:40,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:45,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:46,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:52:46,449 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:52:47,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:51,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:52:57,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:52:57,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:01,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:53:01,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:53:02,020 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:53:02,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:07,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:08,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:13,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:16,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:53:16,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:53:17,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:19,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:24,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:27,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:30,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:31,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:53:31,450 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:53:32,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:53:35,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:37,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:41,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:46,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:53:46,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:53:47,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:47,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:52,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:53:57,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:53:58,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:00,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:54:00,857 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:54:00,858 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:54:00,858 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:54:01,450 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:54:01,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:54:01,493 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:54:02,025 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:54:03,494 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:54:03,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:07,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:09,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:14,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:17,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:17,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:54:17,818 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:54:20,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:25,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:27,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:30,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:32,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:54:32,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:54:32,813 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:54:35,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:37,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:41,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:46,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:47,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:54:47,813 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:54:47,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:51,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:54:57,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:54:57,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:02,033 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:55:02,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:55:02,814 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:55:03,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:07,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:08,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:13,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:17,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:55:17,814 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:55:17,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:19,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:24,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:27,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:30,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:32,033 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:55:32,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:55:32,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:55:36,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:37,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:41,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:46,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:47,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:55:47,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:55:47,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:51,969 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:55:57,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:55:57,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:02,036 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:56:02,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:56:02,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:56:02,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:03,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:56:03,855 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:56:03,855 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:56:03,856 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:56:04,551 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:56:05,551 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:56:07,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:08,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:13,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:17,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:56:17,815 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:56:17,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:19,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:24,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:27,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:30,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:32,037 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:56:32,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:56:32,816 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:56:36,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:37,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:41,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:46,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:47,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:56:47,816 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:56:47,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:51,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:56:57,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:56:57,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:02,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:57:02,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:57:02,829 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:57:02,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:07,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:08,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:14,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:17,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:17,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:57:17,820 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:57:20,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:25,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:27,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:31,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:32,042 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:57:32,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:57:32,819 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:57:36,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:37,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:42,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:47,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:57:47,820 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:57:47,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:48,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:53,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:57:57,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:57:59,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:02,044 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:58:02,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:58:02,820 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:58:05,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:05,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 18:58:05,203 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 18:58:05,204 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 18:58:05,205 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 18:58:05,601 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 18:58:07,602 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 18:58:07,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:15,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:17,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:58:17,820 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:58:17,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:21,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:26,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:27,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:32,047 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:58:32,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:32,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:58:32,821 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:58:37,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:37,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:42,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:47,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:47,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:58:47,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:58:48,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:53,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:58:57,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:58:59,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:02,050 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:59:02,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:59:02,822 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:59:05,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:07,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:59:10,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:16,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:17,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:59:17,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:59:17,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:59:21,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:27,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:59:27,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:32,055 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 18:59:32,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:59:32,822 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:59:33,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:37,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:59:38,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:43,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:47,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 18:59:47,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 18:59:47,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 18:59:48,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:54,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 18:59:57,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:00,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:02,054 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:00:02,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:00:02,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:00:05,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:06,563 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:00:06,565 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:00:06,566 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:00:06,569 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:00:06,660 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:00:07,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:09,662 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:00:11,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:16,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:17,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:00:17,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:00:17,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:22,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:27,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:27,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:32,057 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:00:32,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:00:32,823 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:00:32,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:37,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:38,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:43,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:47,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:00:47,824 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:00:47,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:48,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:54,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:00:57,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:00:59,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:02,060 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:01:02,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:01:02,824 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:01:05,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:07,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:01:11,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:16,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:17,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:01:17,825 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:01:17,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:01:22,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:27,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:01:28,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:32,062 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:01:32,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:01:32,825 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:01:34,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:37,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:01:39,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:45,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:47,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:01:47,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:01:47,826 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:01:51,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:56,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:01:57,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:01,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:02,064 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:02:02,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:02:02,826 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:02:07,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:07,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:09,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:02:09,319 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:02:09,319 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:02:09,320 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:02:09,716 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:02:10,717 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:02:13,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:17,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:02:17,826 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:02:17,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:18,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:23,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:27,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:28,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:32,068 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:02:32,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:02:32,827 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:02:34,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:37,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:39,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:45,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:47,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:02:47,828 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:02:47,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:02:51,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:56,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:02:57,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:01,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:02,070 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:03:02,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:03:02,829 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:03:07,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:07,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:13,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:17,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:03:17,828 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:03:17,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:18,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:24,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:27,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:29,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:32,072 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:03:32,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:03:32,829 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:03:35,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:37,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:41,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:46,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:47,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:03:47,828 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:03:47,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:52,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:03:57,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:03:57,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:02,078 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:04:02,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:04:02,829 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:04:03,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:07,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:08,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:10,472 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:04:10,474 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:04:10,474 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:04:10,475 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:04:10,768 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:04:12,769 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:04:13,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:17,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:17,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:04:17,852 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:04:19,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:24,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:27,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:30,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:32,078 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:04:32,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:04:32,840 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:04:35,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:37,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:40,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:46,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:47,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:04:47,840 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:04:47,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:52,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:04:57,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:04:58,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:02,082 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:05:02,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:05:02,840 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:05:04,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:07,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:05:10,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:15,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:17,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:05:17,841 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:05:17,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:05:21,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:27,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:27,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:05:32,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:32,083 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:05:32,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:05:32,840 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:05:37,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:37,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:05:43,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:47,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:05:47,841 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:05:47,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:05:49,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:54,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:05:57,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:00,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:02,085 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:06:02,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:06:02,841 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:06:05,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:07,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:10,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:11,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:06:11,585 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:06:11,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:06:11,587 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:06:11,817 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:06:12,818 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:06:16,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:17,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:06:17,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:06:17,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:22,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:27,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:27,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:32,088 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:06:32,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:32,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:06:32,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:06:37,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:38,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:44,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:47,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:06:47,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:06:47,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:49,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:54,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:06:57,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:06:59,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:02,090 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:07:02,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:07:02,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:07:05,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:07,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:07:11,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:17,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:17,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:07:17,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:07:17,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:07:23,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:27,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:07:29,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:32,093 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:07:32,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:07:32,842 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:07:34,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:37,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:07:40,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:45,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:47,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:07:47,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:07:47,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:07:50,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:56,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:07:57,866 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:01,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:02,095 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:08:02,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:08:02,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:08:06,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:07,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:12,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:14,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:08:14,364 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:08:14,365 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:08:14,365 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:08:14,889 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:08:14,890 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:08:17,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:17,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:08:17,845 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:08:17,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:23,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:27,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:29,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:32,100 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:08:32,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:08:32,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:08:34,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:37,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:39,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:44,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:47,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:08:47,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:08:47,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:08:50,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:55,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:08:57,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:00,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:02,099 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:09:02,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:09:02,846 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:09:06,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:07,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:11,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:16,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:17,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:09:17,847 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:09:17,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:21,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:26,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:27,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:32,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:32,103 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:09:32,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:09:32,848 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:09:37,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:38,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:44,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:47,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:09:47,847 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:09:47,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:09:50,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:55,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:09:57,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:01,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:02,104 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:10:02,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:10:02,847 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:10:07,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:07,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:12,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:15,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:10:15,179 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:10:15,179 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:10:15,180 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:10:15,951 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:10:16,951 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:10:17,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:17,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:10:17,848 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:10:17,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:23,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:27,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:29,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:32,106 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:10:32,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:10:32,849 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:10:34,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:37,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:40,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:45,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:47,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:10:47,849 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:10:47,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:10:51,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:56,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:10:57,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:01,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:02,108 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:11:02,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:11:02,850 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:11:06,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:07,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:12,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:17,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:11:17,850 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:11:17,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:18,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:23,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:27,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:29,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:32,110 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:11:32,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:11:32,850 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:11:34,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:37,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:39,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:44,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:47,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:11:47,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:11:47,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:11:50,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:55,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:11:57,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:00,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:02,113 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:12:02,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:12:02,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:12:05,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:07,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:11,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:16,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:16,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:12:16,112 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:12:16,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:12:16,113 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:12:17,018 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:12:17,019 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:12:17,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:12:17,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:12:17,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:22,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:27,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:27,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:32,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:12:32,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:32,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:12:32,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:12:37,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:38,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:44,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:47,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:12:47,851 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:12:47,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:49,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:54,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:12:57,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:12:59,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:02,117 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:13:02,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:13:02,852 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:13:04,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:07,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:10,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:16,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:17,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:13:17,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:13:17,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:21,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:26,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:27,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:32,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:32,122 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:13:32,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:13:32,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:13:37,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:38,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:43,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:47,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:13:47,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:13:48,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:48,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:54,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:13:58,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:13:59,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:02,122 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:14:02,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:14:02,861 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:14:04,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:08,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:10,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:16,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:17,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:14:17,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:14:18,572 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:14:18,574 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:14:18,574 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:14:18,576 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:14:18,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:19,074 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:14:19,074 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:14:22,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:27,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:28,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:32,124 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:14:32,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:32,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:14:32,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:14:38,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:38,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:43,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:47,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:14:47,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:14:48,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:49,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:54,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:14:58,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:14:59,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:02,128 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:15:02,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:15:02,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:15:05,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:08,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:10,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:15,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:17,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:15:17,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:15:18,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:21,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:27,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:28,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:32,128 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:15:32,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:32,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:15:32,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:15:38,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:38,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:44,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:47,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:15:47,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:15:48,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:49,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:54,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:15:58,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:15:59,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:02,131 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:16:02,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:16:02,863 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:16:04,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:08,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:16:09,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:15,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:17,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:16:17,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:16:18,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:16:19,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:16:19,447 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:16:19,447 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:16:19,448 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:16:20,126 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:16:20,487 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:21,127 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:16:26,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:28,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:16:32,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:32,133 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:16:32,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:16:32,864 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:16:37,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:38,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:16:43,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:47,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:16:47,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:16:48,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:16:49,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:54,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:16:58,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:00,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:02,136 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:17:02,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:17:02,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:17:05,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:08,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:11,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:16,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:17,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:17:17,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:17:18,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:22,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:27,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:28,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:32,138 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:17:32,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:32,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:17:32,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:17:38,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:38,868 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:43,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:47,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:17:47,874 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:17:48,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:17:49,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:54,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:17:58,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:00,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:02,145 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:18:02,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:18:02,874 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:18:05,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:08,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:11,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:17,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:17,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:18:17,874 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:18:18,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:20,449 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:18:20,450 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:18:20,451 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:18:20,451 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:18:21,185 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:18:21,185 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:18:22,491 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:28,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:28,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:32,143 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:18:32,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:18:32,875 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:18:34,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:38,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:39,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:44,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:47,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:18:47,875 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:18:48,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:18:49,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:54,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:18:58,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:00,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:02,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:19:02,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:19:02,875 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:19:06,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:08,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:12,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:17,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:17,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:19:17,875 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:19:18,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:23,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:28,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:28,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:32,148 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:19:32,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:19:32,875 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:19:33,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:38,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:39,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:44,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:47,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:19:47,876 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:19:48,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:19:50,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:55,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:19:58,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:01,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:02,150 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:20:02,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:20:02,878 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:20:07,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:08,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:12,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:17,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:20:17,878 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:20:18,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:18,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:23,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:20:23,027 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:20:23,027 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:20:23,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:23,028 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:20:23,252 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:20:25,252 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:20:28,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:28,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:32,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:20:32,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:20:32,885 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:20:33,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:38,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:39,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:45,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:47,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:20:47,885 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:20:48,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:20:50,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:56,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:20:58,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:02,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:02,154 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:21:02,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:21:02,885 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:21:07,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:08,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:13,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:17,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:21:17,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:21:18,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:18,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:24,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:28,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:30,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:32,156 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:21:32,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:21:32,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:21:36,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:38,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:41,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:46,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:47,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:21:47,886 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:21:48,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:21:52,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:57,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:21:58,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:02,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:02,158 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:22:02,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:22:02,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:22:08,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:08,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:13,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:17,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:22:17,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:22:18,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:18,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:23,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:22:23,948 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:22:23,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:22:23,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:23,950 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:22:24,300 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:22:25,301 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:22:28,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:29,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:32,164 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:22:32,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:22:32,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:22:35,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:38,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:40,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:46,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:47,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:22:47,887 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:22:48,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:22:51,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:56,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:22:58,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:01,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:02,163 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:23:02,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:23:02,900 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:23:07,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:08,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:12,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:17,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:23:17,900 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:23:18,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:18,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:24,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:28,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:30,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:32,168 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:23:32,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:23:32,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:23:35,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:38,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:41,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:46,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:47,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:23:47,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:23:48,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:23:52,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:58,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:23:58,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:02,170 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:24:02,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:24:02,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:24:03,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:08,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:09,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:15,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:17,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:24:17,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:24:18,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:20,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:25,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:25,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:24:25,363 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:24:25,364 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:24:25,366 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:24:26,357 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:24:27,357 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:24:28,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:31,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:32,173 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:24:32,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:24:32,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:24:36,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:38,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:41,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:47,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:47,901 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:24:47,902 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:24:48,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:24:53,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:58,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:24:58,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:02,177 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:25:02,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:25:02,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:25:03,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:08,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:09,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:15,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:17,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:25:17,903 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:25:18,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:21,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:26,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:28,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:32,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:32,178 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:25:32,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:25:32,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:25:37,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:38,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:42,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:47,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:25:47,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:25:48,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:48,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:25:54,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:25:58,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:00,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:02,179 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:26:02,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:26:02,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:26:05,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:08,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:10,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:16,105 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:17,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:26:17,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:26:18,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:21,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:26,446 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:26:26,447 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:26:26,448 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:26:26,448 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:26,449 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:26:27,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:26:27,415 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:26:28,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:32,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:32,181 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:26:32,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:26:32,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:26:37,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:38,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:42,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:47,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:26:47,905 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:26:48,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:48,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:26:53,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:58,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:26:58,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:02,188 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:27:02,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:27:02,905 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:27:04,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:08,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:09,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:15,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:17,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:27:17,905 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:27:18,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:20,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:25,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:28,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:31,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:32,186 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:27:32,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:27:32,905 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:27:36,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:38,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:42,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:47,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:27:47,906 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:27:48,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:48,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:27:53,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:58,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:27:58,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:02,190 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:28:02,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:28:02,906 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:28:03,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:08,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:09,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:15,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:17,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:28:17,906 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:28:18,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:21,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:26,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:28,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:29,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:28:29,314 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:28:29,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:28:29,315 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:28:29,474 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:28:31,475 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:28:32,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:32,190 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:28:32,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:28:32,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:28:37,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:38,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:43,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:47,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:28:47,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:28:48,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:48,890 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:28:54,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:28:58,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:00,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:02,194 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:29:02,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:29:02,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:29:06,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:08,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:11,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:16,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:17,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:29:17,907 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:29:18,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:22,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:28,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:28,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:32,195 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:29:32,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:29:32,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:29:34,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:38,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:39,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:44,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:47,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:29:47,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:29:48,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:29:50,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:55,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:29:58,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:01,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:02,198 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:30:02,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:30:02,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:30:07,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:08,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:13,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:17,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:30:17,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:30:18,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:19,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:24,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:28,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:29,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:30,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:30:30,862 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:30:30,862 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:30:30,863 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:30:31,533 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:30:32,201 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:30:32,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:30:32,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:30:33,534 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:30:35,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:38,895 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:40,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:46,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:47,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:30:47,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:30:48,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:30:52,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:57,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:30:58,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:02,203 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:31:02,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:02,908 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:31:02,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:31:08,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:08,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:13,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:17,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:31:17,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:31:18,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:18,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:24,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:28,897 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:29,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:32,209 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:31:32,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:31:32,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:31:35,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:38,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:40,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:45,129 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:47,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:31:47,909 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:31:48,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:31:50,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:56,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:31:58,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:02,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:02,208 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:32:02,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:32:02,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:32:08,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:08,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:14,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:17,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:32:17,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:32:18,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:20,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:25,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:28,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:30,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:32,212 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:32:32,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:32:32,560 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:32:32,560 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:32:32,560 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:32:32,593 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:32:32,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:32:32,910 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:32:33,594 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:32:35,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:38,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:41,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:47,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:47,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:32:47,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:32:48,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:32:53,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:59,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:32:59,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:02,212 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:33:02,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:33:02,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:33:05,132 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:09,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:10,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:15,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:17,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:33:17,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:33:19,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:21,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:26,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:29,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:31,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:32,215 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:33:32,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:33:32,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:33:36,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:39,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:42,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:47,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:47,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:33:47,911 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:33:49,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:33:53,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:58,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:33:59,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:02,219 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:34:02,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:34:02,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:34:05,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:09,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:11,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:16,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:17,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:34:17,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:34:19,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:22,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:27,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:29,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:32,219 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:34:32,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:32,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:34:32,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:34:35,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:34:35,558 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:34:35,558 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:34:35,559 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:34:35,644 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:34:37,599 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:37,645 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:34:39,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:43,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:47,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:34:47,912 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:34:49,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:49,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:34:54,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:59,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:34:59,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:02,222 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:35:02,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:35:02,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:35:05,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:09,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:10,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:15,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:17,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:35:17,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:35:19,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:21,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:26,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:29,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:32,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:32,224 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:35:32,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:35:32,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:35:38,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:39,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:43,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:47,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:35:47,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:35:49,130 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:49,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:35:54,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:59,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:35:59,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:02,230 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:36:02,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:36:02,915 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:36:05,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:09,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:10,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:15,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:17,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:36:17,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:36:19,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:21,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:26,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:29,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:32,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:32,229 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:36:32,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:36:32,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:36:37,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:36:37,054 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:36:37,054 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:36:37,057 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:36:37,701 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:36:38,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:39,702 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:36:39,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:43,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:47,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:36:47,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:36:49,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:49,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:36:54,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:59,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:36:59,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:02,232 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:37:02,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:37:02,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:37:05,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:09,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:10,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:15,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:17,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:37:17,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:37:19,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:21,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:26,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:29,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:32,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:32,235 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:37:32,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:37:32,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:37:37,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:39,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:43,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:47,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:37:47,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:37:49,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:49,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:37:54,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:59,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:37:59,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:02,236 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:38:02,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:38:02,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:38:05,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:09,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:10,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:15,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:17,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:38:17,917 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:38:19,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:20,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:26,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:29,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:32,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:32,238 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:38:32,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:38:32,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:38:37,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:38,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:38:38,165 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:38:38,166 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:38:38,167 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:38:38,756 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:38:39,756 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:38:39,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:42,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:47,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:47,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:38:47,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:38:49,960 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:38:53,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:58,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:38:59,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:02,243 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:39:02,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:39:02,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:39:04,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:09,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:09,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:15,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:17,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:39:17,918 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:39:19,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:21,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:26,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:29,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:32,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:32,245 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:39:32,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:39:32,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:39:38,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:39,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:44,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:47,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:39:47,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:39:49,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:39:50,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:55,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:39:59,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:01,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:02,249 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:40:02,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:40:02,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:40:07,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:09,961 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:13,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:17,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:40:17,920 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:40:19,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:19,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:24,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:29,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:29,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:32,256 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:40:32,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:40:32,920 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:40:34,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:39,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:40,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:41,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:40:41,122 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:40:41,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:40:41,123 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:40:41,823 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:40:43,824 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:40:46,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:47,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:40:47,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:40:49,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:40:52,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:57,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:40:59,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:02,256 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:41:02,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:02,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:41:02,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:41:08,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:09,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:13,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:17,928 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:41:17,929 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:41:19,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:19,963 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:24,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:29,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:30,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:32,260 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:41:32,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:41:32,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:41:35,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:39,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:40,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:46,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:47,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:41:47,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:41:49,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:41:51,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:57,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:41:59,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:02,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:02,262 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:42:02,929 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:42:02,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:42:07,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:09,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:13,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:17,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:42:17,930 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:42:19,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:19,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:24,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:29,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:29,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:32,264 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:42:32,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:42:32,931 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:42:34,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:39,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:40,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:42,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:42:42,656 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:42:42,656 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:42:42,657 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:42:42,874 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:42:43,874 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:42:45,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:47,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:42:47,931 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:42:49,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:42:51,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:56,174 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:42:59,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:01,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:02,267 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:43:02,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:43:02,931 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:43:07,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:09,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:13,175 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:17,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:43:17,932 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:43:19,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:19,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:24,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:29,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:30,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:32,269 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:43:32,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:43:32,932 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:43:35,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:39,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:41,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:46,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:47,931 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:43:47,932 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:43:49,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:43:52,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:57,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:43:59,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:02,271 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:44:02,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:02,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:44:02,940 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:44:08,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:09,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:14,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:17,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:44:17,940 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:44:19,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:20,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:25,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:29,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:30,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:32,274 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:44:32,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:44:32,940 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:44:36,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:39,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:42,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:44,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:44:44,128 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:44:44,129 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:44:44,130 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:44:44,924 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:44:45,925 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:44:47,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:44:47,940 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:44:48,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:49,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:44:53,180 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:58,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:44:59,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:02,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:45:02,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:45:02,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:45:04,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:09,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:09,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:14,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:17,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:45:17,941 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:45:19,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:20,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:25,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:29,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:31,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:32,278 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:45:32,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:45:32,942 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:45:37,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:39,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:42,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:47,940 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:45:47,941 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:45:48,094 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:49,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:45:53,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:58,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:45:59,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:02,282 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:46:02,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:46:02,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:46:04,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:09,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:09,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:15,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:17,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:46:17,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:46:19,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:21,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:26,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:29,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:32,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:32,283 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:46:32,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:46:32,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:46:37,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:39,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:42,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:47,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:46:47,268 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:46:47,269 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:46:47,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:47,270 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:46:47,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:46:47,942 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:46:47,984 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:46:49,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:46:49,985 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:46:53,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:59,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:46:59,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:02,285 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:47:02,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:47:02,942 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:47:05,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:09,983 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:10,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:15,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:17,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:47:17,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:47:19,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:21,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:26,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:29,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:32,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:32,287 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:47:32,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:47:32,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:47:37,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:39,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:43,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:47,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:47:47,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:47:49,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:49,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:47:54,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:59,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:47:59,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:02,290 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:48:02,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:48:02,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:48:04,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:09,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:10,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:15,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:17,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:48:17,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:48:19,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:21,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:27,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:29,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:32,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:48:32,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:32,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:48:32,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:48:38,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:39,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:44,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:47,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:48:47,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:48:48,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:48:48,695 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:48:48,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:48:48,696 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:48:49,047 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:48:49,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:49,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:48:50,048 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:48:55,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:48:59,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:01,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:02,294 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:49:02,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:49:02,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:49:06,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:09,989 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:12,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:17,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:17,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:49:17,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:49:19,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:23,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:28,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:30,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:32,300 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:49:32,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:49:32,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:49:34,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:39,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:39,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:45,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:47,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:49:47,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:49:49,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:49:50,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:56,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:49:59,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:02,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:02,299 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:50:02,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:50:02,946 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:50:08,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:09,992 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:13,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:17,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:50:17,946 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:50:19,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:19,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:24,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:29,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:29,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:32,310 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:50:32,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:50:32,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:50:35,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:39,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:40,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:45,208 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:47,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:50:47,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:50:49,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:50:50,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:50:50,524 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:50:50,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:50:50,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:50,526 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:50:51,112 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:50:52,114 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:50:55,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:50:58,118 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:50:59,993 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:01,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:02,304 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:51:02,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:51:02,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:51:06,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:08,127 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:51:09,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:12,177 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:17,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:17,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:51:17,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:51:19,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:22,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:28,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:29,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:32,306 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:51:32,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:51:32,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:51:33,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:39,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:39,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:44,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:47,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:51:47,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:51:49,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:51:50,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:55,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:51:59,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:00,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:02,308 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:52:02,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:52:02,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:52:06,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:09,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:11,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:16,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:17,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:52:17,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:52:19,995 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:21,210 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:27,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:29,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:32,310 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:52:32,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:34,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:52:34,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:52:38,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:39,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:44,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:49,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:49,518 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:52:49,519 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:52:49,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:52:55,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:52:59,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:01,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:02,313 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:53:04,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:53:04,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:53:06,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:08,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:53:08,759 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:53:08,760 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:53:08,761 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:53:09,197 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:53:09,996 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:10,198 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:53:11,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:17,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:19,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:53:19,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:53:19,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:22,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:28,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:29,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:32,316 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:53:33,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:34,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:53:34,520 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:53:39,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:39,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:44,192 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:49,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:49,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:53:49,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:53:49,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:53:55,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:53:59,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:01,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:02,322 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:54:04,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:54:04,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:54:06,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:09,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:12,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:17,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:19,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:54:19,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:54:19,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:22,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:27,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:29,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:32,321 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:54:32,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:34,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:54:34,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:54:38,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:39,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:44,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:49,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:54:49,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:54:49,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:49,998 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:54:55,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:54:59,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:00,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:02,325 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:55:04,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:55:04,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:55:05,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:09,923 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:55:09,925 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:55:09,925 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:55:09,926 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:55:09,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:10,260 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:55:10,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:12,261 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:55:16,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:19,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:55:19,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:55:19,999 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:21,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:27,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:30,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:32,327 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:55:32,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:34,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:55:34,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:55:38,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:40,000 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:43,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:49,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:49,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:55:49,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:55:50,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:55:54,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:55:59,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:00,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:02,328 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:56:04,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:04,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:56:04,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:56:10,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:10,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:15,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:19,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:56:19,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:56:20,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:20,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:26,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:30,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:32,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:32,330 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:56:34,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:56:34,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:56:37,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:40,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:42,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:47,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:49,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:56:49,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:56:50,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:56:52,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:56:57,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:00,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:02,333 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:57:02,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:04,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:57:04,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:57:08,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:10,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:11,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:57:11,245 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:57:11,245 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:57:11,246 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:57:11,330 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:57:12,330 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:57:13,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:19,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:19,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:57:19,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:57:20,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:25,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:30,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:30,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:32,335 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:57:34,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:57:34,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:57:35,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:40,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:41,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:47,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:49,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:57:49,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:57:50,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:57:52,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:57:57,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:00,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:02,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:02,338 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:58:04,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:58:04,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:58:08,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:10,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:14,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:19,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:19,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:58:19,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:58:20,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:24,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:30,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:30,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:32,344 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:58:34,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:58:34,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:58:35,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:40,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:41,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:46,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:49,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:58:49,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:58:50,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:58:51,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:58:57,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:00,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:02,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:02,342 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:59:04,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:59:04,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:59:08,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:10,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:14,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 19:59:14,100 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 19:59:14,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 19:59:14,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:14,103 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 19:59:14,396 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 19:59:16,396 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 19:59:19,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:19,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:59:19,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:59:20,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:24,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:29,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:30,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:32,344 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 19:59:34,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:34,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:59:34,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:59:40,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:40,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:45,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:49,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 19:59:49,539 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 19:59:50,366 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 19:59:50,719 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 19:59:56,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:00,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:01,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:02,357 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:00:04,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:00:04,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:00:06,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:10,371 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:12,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:17,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:19,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:00:19,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:00:20,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:22,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:28,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:30,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:32,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:00:33,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:34,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:00:34,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:00:39,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:40,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:45,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:49,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:00:49,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:00:50,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:00:50,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:00:56,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:00,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:01,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:02,353 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:01:04,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:01:04,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:01:06,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:10,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:12,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:15,562 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:01:15,563 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:01:15,564 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:01:15,564 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:01:16,457 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:01:16,458 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:01:17,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:19,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:01:19,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:01:20,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:22,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:28,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:30,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:32,355 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:01:33,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:34,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:01:34,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:01:39,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:40,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:45,242 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:49,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:01:49,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:01:50,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:01:50,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:01:56,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:00,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:02,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:02,358 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:02:04,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:02:04,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:02:08,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:10,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:13,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:18,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:19,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:02:19,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:02:20,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:24,244 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:29,245 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:30,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:32,360 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:02:34,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:34,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:02:34,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:02:40,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:40,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:46,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:49,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:02:49,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:02:50,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:02:51,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:02:57,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:00,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:02,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:02,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:03:04,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:03:04,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:03:08,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:10,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:14,250 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:16,470 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:03:16,472 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:03:16,472 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:03:16,473 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:03:16,516 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:03:18,517 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:03:19,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:19,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:03:19,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:03:20,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:25,251 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:30,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:30,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:32,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:03:34,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:03:34,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:03:35,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:40,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:41,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:46,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:49,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:03:49,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:03:50,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:03:51,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:03:57,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:00,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:02,369 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:04:02,370 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:04,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:04:04,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:04:08,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:10,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:13,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:19,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:19,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:04:19,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:04:20,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:24,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:30,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:30,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:32,369 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:04:34,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:04:34,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:04:35,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:40,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:41,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:47,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:49,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:04:49,544 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:04:50,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:04:52,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:04:57,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:00,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:02,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:02,371 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:05:04,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:05:04,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:05:07,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:10,385 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:12,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:17,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:05:17,918 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:05:17,919 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:05:17,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:17,920 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:05:18,586 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:05:19,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:05:19,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:05:20,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:20,587 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:05:23,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:29,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:30,386 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:32,374 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:05:34,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:34,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:05:34,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:05:40,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:40,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:45,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:49,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:05:49,545 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:05:50,387 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:05:50,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:05:56,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:00,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:01,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:02,376 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:06:04,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:06:04,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:06:06,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:10,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:12,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:18,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:19,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:06:19,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:06:20,388 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:23,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:29,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:30,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:32,378 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:06:34,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:34,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:06:34,547 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:06:40,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:40,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:46,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:49,547 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:06:49,548 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:06:50,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:06:51,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:06:57,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:00,390 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:02,380 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:07:02,382 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:04,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:07:04,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:07:08,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:10,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:14,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:19,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:19,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:07:19,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:07:20,391 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:20,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:07:20,685 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:07:20,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:07:20,687 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:07:21,656 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:07:22,657 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:07:24,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:29,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:30,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:32,388 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:07:34,389 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:34,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:07:34,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:07:40,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:40,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:45,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:49,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:07:49,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:07:50,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:07:50,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:07:56,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:00,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:01,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:02,385 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:08:04,548 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:08:04,549 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:08:06,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:10,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:12,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:17,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:19,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:08:19,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:08:20,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:22,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:28,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:30,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:32,389 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:08:33,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:34,549 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:08:34,550 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:08:39,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:40,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:45,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:49,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:08:49,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:08:50,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:08:50,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:08:56,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:00,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:02,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:02,390 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:09:04,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:09:04,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:09:07,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:10,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:12,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:18,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:19,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:09:19,551 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:09:20,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:22,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:09:22,415 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:09:22,416 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:09:22,416 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:09:22,720 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:09:23,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:24,722 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:09:29,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:30,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:32,393 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:09:34,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:34,551 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:09:34,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:09:40,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:40,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:46,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:49,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:09:49,552 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:09:50,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:09:52,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:09:57,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:00,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:02,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:02,395 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:10:04,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:10:04,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:10:07,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:10,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:12,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:18,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:19,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:10:19,553 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:10:20,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:23,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:28,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:30,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:32,397 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:10:33,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:34,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:10:34,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:10:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:40,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:45,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:49,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:10:49,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:10:50,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:10:50,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:10:56,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:00,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:02,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:02,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:11:04,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:11:04,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:11:08,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:10,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:13,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:19,311 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:19,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:11:19,554 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:11:20,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:24,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:11:24,035 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:11:24,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:11:24,036 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:11:24,782 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:11:25,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:26,782 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:11:30,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:30,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:32,403 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:11:34,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:11:34,557 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:11:35,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:40,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:41,317 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:46,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:50,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:11:50,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:11:50,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:11:51,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:11:57,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:00,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:02,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:02,413 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:12:05,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:12:05,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:12:07,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:10,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:12,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:17,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:20,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:20,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:12:20,521 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:12:23,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:29,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:30,520 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:32,407 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:12:34,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:35,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:12:35,522 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:12:40,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:40,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:45,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:50,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:12:50,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:12:50,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:12:50,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:12:56,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:00,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:01,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:02,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:13:05,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:13:05,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:13:06,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:10,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:12,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:17,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:20,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:20,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:13:20,523 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:13:23,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:26,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:13:26,603 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:13:26,604 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:13:26,605 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:13:26,842 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:13:28,843 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:13:29,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:30,522 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:32,412 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:13:34,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:35,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:13:35,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:13:40,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:40,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:45,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:50,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:13:50,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:13:50,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:13:50,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:13:56,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:00,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:01,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:02,414 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:14:05,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:14:05,524 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:14:06,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:10,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:12,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:17,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:20,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:14:20,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:14:20,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:22,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:28,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:30,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:32,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:14:33,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:35,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:14:35,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:14:39,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:40,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:45,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:50,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:14:50,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:14:50,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:14:50,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:14:56,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:00,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:01,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:02,418 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:15:05,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:15:05,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:15:06,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:10,552 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:12,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:17,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:20,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:15:20,525 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:15:20,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:22,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:27,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:27,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:15:27,609 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:15:27,609 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:15:27,610 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:15:27,892 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:15:29,893 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:15:30,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:32,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:32,421 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:15:35,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:15:35,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:15:38,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:40,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:44,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:50,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:15:50,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:15:50,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:15:50,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:15:56,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:00,553 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:02,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:02,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:16:05,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:16:05,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:16:08,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:10,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:14,359 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:19,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:20,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:16:20,526 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:16:20,566 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:25,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:30,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:31,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:32,425 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:16:35,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:16:35,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:16:36,727 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:40,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:42,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:48,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:50,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:16:50,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:16:50,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:16:54,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:16:59,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:00,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:02,431 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:17:04,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:05,526 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:17:05,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:17:10,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:10,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:16,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:20,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:17:20,527 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:17:20,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:21,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:27,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:28,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:17:28,618 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:17:28,618 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:17:28,619 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:17:28,945 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:17:29,946 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:17:30,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:32,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:32,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:17:35,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:17:35,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:17:37,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:40,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:42,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:48,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:50,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:17:50,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:17:50,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:17:54,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:17:59,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:00,556 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:02,434 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:18:04,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:05,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:18:05,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:18:10,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:10,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:15,845 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:20,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:18:20,528 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:18:20,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:21,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:26,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:30,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:32,435 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:18:32,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:35,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:18:35,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:18:37,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:40,557 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:42,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:47,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:50,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:18:50,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:18:50,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:18:53,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:18:58,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:00,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:02,437 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:19:03,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:05,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:19:05,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:19:09,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:10,558 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:14,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:19,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:20,528 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:19:20,529 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:19:20,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:24,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:30,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:30,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:31,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:19:31,034 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:19:31,035 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:19:31,035 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:19:32,023 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:19:32,023 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:19:32,439 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:19:35,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:19:35,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:19:36,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:40,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:41,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:47,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:50,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:19:50,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:19:50,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:19:53,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:19:58,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:00,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:02,442 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:20:03,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:05,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:20:05,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:20:09,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:10,579 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:15,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:20,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:20:20,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:20:20,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:21,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:26,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:30,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:32,444 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:20:32,447 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:35,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:20:35,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:20:37,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:40,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:42,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:48,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:50,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:20:50,530 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:20:50,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:20:54,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:20:59,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:00,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:02,447 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:21:05,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:21:05,531 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:21:05,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:10,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:15,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:20,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:21:20,531 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:21:20,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:21,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:26,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:30,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:32,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:21:32,135 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:21:32,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:21:32,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:32,137 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:21:32,454 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:21:33,094 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:21:34,094 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:21:35,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:21:35,532 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:21:37,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:40,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:42,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:48,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:50,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:21:50,532 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:21:50,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:21:54,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:21:59,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:00,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:02,452 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:22:05,531 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:22:05,532 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:22:05,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:10,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:10,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:15,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:20,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:22:20,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:22:20,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:21,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:26,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:30,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:32,456 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:22:32,457 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:35,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:22:35,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:22:37,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:40,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:42,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:48,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:50,532 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:22:50,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:22:50,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:22:54,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:22:59,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:00,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:02,457 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:23:05,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:23:05,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:23:05,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:10,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:10,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:16,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:20,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:23:20,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:23:20,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:22,728 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:27,860 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:30,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:32,460 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:23:33,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:23:33,153 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:23:33,153 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:23:33,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:33,155 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:23:33,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:23:34,156 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:23:35,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:23:35,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:23:38,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:40,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:43,861 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:49,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:50,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:23:50,534 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:23:50,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:23:54,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:23:59,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:00,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:02,462 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:24:05,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:24:05,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:24:05,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:10,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:10,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:16,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:20,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:24:20,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:24:20,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:22,711 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:27,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:30,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:32,464 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:24:33,465 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:35,534 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:24:35,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:24:38,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:40,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:44,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:49,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:24:50,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:24:50,535 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:24:50,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:24:55,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:00,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:01,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:02,466 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:25:05,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:25:05,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:25:06,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:10,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:12,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:18,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:20,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:25:20,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:25:20,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:23,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:29,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:30,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:32,468 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:25:35,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:35,535 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:25:35,536 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:25:35,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:25:35,830 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:25:35,830 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:25:35,831 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:25:36,210 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:25:38,211 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:25:40,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:41,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:46,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:50,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:25:50,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:25:50,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:25:51,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:25:57,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:00,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:02,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:02,475 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:26:05,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:26:05,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:26:07,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:10,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:13,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:18,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:20,536 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:26:20,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:26:20,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:23,729 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:29,006 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:30,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:32,474 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:26:35,007 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:35,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:26:35,537 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:26:40,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:41,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:47,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:50,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:26:50,538 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:26:50,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:26:52,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:26:58,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:00,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:02,477 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:27:03,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:05,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:27:05,538 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:27:08,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:10,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:14,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:19,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:20,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:27:20,538 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:27:20,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:24,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:29,009 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:30,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:32,478 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:27:35,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:35,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:27:35,539 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:27:36,892 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:27:36,894 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:27:36,895 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:27:36,895 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:27:37,272 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:27:38,272 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:27:40,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:41,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:46,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:50,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:27:50,539 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:27:50,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:27:51,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:27:57,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:00,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:02,481 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:28:02,483 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:05,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:28:05,539 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:28:07,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:10,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:13,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:19,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:20,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:28:20,539 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:28:20,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:25,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:30,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:31,013 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:32,483 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:28:35,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:28:35,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:28:36,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:40,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:42,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:48,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:50,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:28:50,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:28:50,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:28:53,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:28:59,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:00,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:02,486 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:29:05,014 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:05,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:29:05,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:29:10,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:11,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:17,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:20,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:29:20,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:29:20,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:22,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:28,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:30,609 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:32,488 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:29:33,490 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:35,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:29:35,540 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:29:38,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:29:38,069 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:29:38,069 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:29:38,070 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:29:38,327 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:29:39,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:40,328 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:29:40,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:44,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:50,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:29:50,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:29:50,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:29:50,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:29:56,017 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:00,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:01,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:02,491 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:30:05,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:30:05,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:30:06,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:10,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:12,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:18,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:20,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:30:20,541 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:30:20,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:23,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:29,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:30,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:32,498 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:30:34,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:35,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:30:35,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:30:40,019 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:40,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:46,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:50,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:30:50,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:30:50,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:30:51,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:30:57,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:00,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:02,495 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:31:02,497 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:06,942 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:31:06,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:31:08,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:10,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:14,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:20,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:20,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:21,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:31:21,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:31:26,020 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:30,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:32,021 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:32,499 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:31:36,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:31:36,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:31:37,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:40,570 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:31:40,571 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:31:40,572 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:31:40,573 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:31:40,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:41,378 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:31:42,378 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:31:42,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:48,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:50,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:31:51,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:31:51,943 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:31:53,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:31:59,022 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:00,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:02,500 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:32:04,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:06,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:32:06,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:32:10,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:10,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:16,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:20,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:21,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:21,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:32:21,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:32:27,024 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:30,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:32,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:32,503 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:32:36,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:32:36,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:32:37,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:40,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:43,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:48,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:50,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:32:51,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:32:51,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:32:53,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:32:59,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:00,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:02,504 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:33:04,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:06,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:33:06,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:33:09,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:10,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:15,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:20,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:21,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:21,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:33:21,944 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:33:26,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:30,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:32,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:32,507 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:33:36,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:33:36,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:33:37,126 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:40,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:41,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:33:41,410 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:33:41,410 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:33:41,411 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:33:41,421 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:33:42,421 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:33:42,451 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:48,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:50,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:33:51,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:33:51,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:33:53,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:33:59,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:00,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:02,510 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:34:05,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:06,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:34:06,945 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:34:10,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:10,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:16,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:20,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:21,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:21,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:34:21,946 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:34:27,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:30,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:32,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:32,512 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:34:36,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:34:36,946 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:34:37,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:40,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:43,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:48,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:50,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:34:51,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:34:51,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:34:53,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:34:59,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:00,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:02,516 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:35:05,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:06,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:35:06,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:35:10,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:11,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:16,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:20,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:21,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:35:21,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:35:22,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:28,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:30,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:32,522 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:35:33,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:36,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:35:36,947 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:35:39,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:40,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:42,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:35:42,354 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:35:42,354 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:35:42,355 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:35:42,472 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:35:44,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:44,473 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:35:49,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:35:50,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:35:51,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:35:51,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:35:55,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:00,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:00,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:02,521 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:36:06,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:06,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:36:06,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:36:10,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:12,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:17,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:20,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:21,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:36:21,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:36:22,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:28,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:30,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:32,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:36:33,527 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:36,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:36:36,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:36:39,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:40,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:44,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:50,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:36:50,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:36:51,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:36:51,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:36:55,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:00,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:01,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:02,525 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:37:06,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:06,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:37:06,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:37:10,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:12,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:17,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:20,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:21,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:37:21,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:37:22,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:28,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:30,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:32,528 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:37:33,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:36,948 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:37:36,949 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:37:39,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:40,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:44,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:45,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:37:45,324 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:37:45,324 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:37:45,325 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:37:45,526 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:37:46,526 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:37:50,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:37:50,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:37:51,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:37:51,950 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:37:56,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:00,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:01,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:02,530 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:38:06,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:06,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:38:06,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:38:10,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:12,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:18,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:20,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:21,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:38:21,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:38:23,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:29,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:30,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:32,532 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:38:35,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:36,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:38:36,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:38:40,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:41,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:46,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:50,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:38:51,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:38:51,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:38:52,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:38:58,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:00,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:02,535 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:39:03,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:06,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:39:06,951 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:39:09,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:10,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:14,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:20,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:20,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:21,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:39:21,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:39:26,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:30,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:31,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:32,537 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:39:36,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:39:36,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:39:37,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:40,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:43,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:46,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:39:46,983 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:39:46,984 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:39:46,985 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:39:47,588 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:39:48,588 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:39:49,027 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:50,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:39:51,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:39:51,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:39:54,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:39:59,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:00,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:02,543 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:40:04,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:06,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:40:06,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:40:10,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:10,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:16,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:20,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:21,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:40:21,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:40:22,171 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:28,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:30,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:32,541 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:40:33,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:36,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:40:36,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:40:39,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:40,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:45,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:50,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:40:50,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:40:51,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:40:51,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:40:56,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:00,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:01,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:02,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:41:06,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:06,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:41:06,953 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:41:10,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:12,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:18,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:20,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:21,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:41:21,955 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:41:23,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:29,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:30,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:32,546 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:41:34,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:36,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:41:36,955 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:41:40,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:40,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:45,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:48,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:41:48,586 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:41:48,586 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:41:48,587 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:41:48,635 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:41:50,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:41:50,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:41:50,636 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:41:51,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:41:51,955 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:41:56,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:00,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:01,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:02,548 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:42:06,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:42:06,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:42:07,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:10,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:13,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:18,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:20,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:21,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:42:21,955 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:42:23,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:29,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:30,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:32,551 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:42:34,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:36,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:42:36,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:42:40,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:40,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:46,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:50,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:42:51,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:42:51,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:42:52,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:42:58,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:00,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:02,553 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:43:03,554 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:06,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:43:06,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:43:09,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:10,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:14,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:20,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:20,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:21,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:43:21,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:43:25,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:30,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:30,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:32,556 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:43:36,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:36,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:43:36,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:43:40,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:42,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:47,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:49,907 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:43:49,908 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:43:49,908 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:43:49,909 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:43:50,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:43:50,681 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:43:51,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:43:51,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:43:52,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:43:52,682 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:43:58,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:00,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:02,558 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:44:03,559 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:06,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:44:06,957 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:44:09,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:10,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:14,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:20,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:20,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:21,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:44:21,957 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:44:26,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:30,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:31,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:32,564 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:44:36,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:36,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:44:36,957 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:44:40,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:42,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:47,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:50,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:44:51,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:44:51,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:44:52,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:44:58,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:00,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:02,563 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:45:03,564 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:06,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:45:06,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:45:09,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:10,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:14,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:19,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:20,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:21,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:45:21,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:45:25,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:30,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:30,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:32,567 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:45:35,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:36,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:45:36,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:45:40,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:41,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:46,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:50,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:45:51,958 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:45:51,958 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:45:52,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:45:52,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:45:52,971 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:45:52,971 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:45:52,972 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:45:53,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:45:54,728 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:45:58,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:00,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:02,568 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:46:03,569 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:06,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:46:06,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:46:09,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:10,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:15,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:20,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:21,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:21,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:46:21,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:46:26,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:30,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:31,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:32,570 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:46:36,975 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:46:36,975 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:46:37,194 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:40,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:43,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:48,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:50,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:46:51,976 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:46:51,976 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:46:53,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:46:59,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:00,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:02,573 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:47:05,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:06,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:47:06,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:47:10,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:11,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:17,084 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:20,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:21,979 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:47:21,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:47:22,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:28,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:30,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:32,575 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:47:33,576 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:36,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:47:36,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:47:39,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:40,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:45,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:50,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:47:51,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:51,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:47:51,980 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:47:54,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:47:54,326 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:47:54,326 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:47:54,327 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:47:54,775 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:47:56,367 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:47:56,776 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:48:00,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:02,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:02,578 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:48:06,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:48:06,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:48:07,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:10,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:13,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:18,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:20,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:21,980 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:48:21,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:48:23,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:29,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:30,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:32,580 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:48:34,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:36,981 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:48:36,981 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:48:40,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:40,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:45,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:50,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:48:50,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:48:51,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:48:51,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:48:55,090 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:00,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:00,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:02,586 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:49:06,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:06,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:49:06,987 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:49:10,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:12,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:17,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:20,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:21,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:49:21,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:49:22,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:28,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:30,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:32,584 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:49:33,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:36,987 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:49:36,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:49:39,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:40,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:44,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:50,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:50,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:49:51,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:49:51,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:49:55,092 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:49:55,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:49:55,665 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:49:55,665 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:49:55,667 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:49:55,833 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:49:56,833 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:50:00,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:01,093 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:02,587 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:50:06,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:50:06,988 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:50:07,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:10,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:13,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:19,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:20,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:22,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:50:22,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:50:25,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:30,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:31,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:32,590 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:50:36,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:37,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:50:37,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:50:40,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:42,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:47,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:50,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:50:52,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:50:52,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:50:52,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:50:58,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:00,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:02,592 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:51:03,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:07,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:51:07,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:51:09,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:10,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:14,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:20,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:20,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:22,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:51:22,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:51:26,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:30,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:32,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:32,594 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:51:37,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:37,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:51:37,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:51:40,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:43,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:48,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:50,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:51:52,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:51:52,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:51:53,496 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:58,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:51:58,658 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:51:58,659 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:51:58,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:51:58,660 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:51:58,896 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:52:00,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:00,897 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:52:02,597 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:52:04,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:07,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:52:07,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:52:09,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:10,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:15,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:20,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:20,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:22,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:52:22,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:52:26,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:30,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:32,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:32,599 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:52:37,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:37,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:52:37,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:52:40,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:43,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:48,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:50,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:52:52,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:52:52,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:52:53,461 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:52:59,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:00,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:02,602 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:53:04,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:07,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:53:07,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:53:10,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:10,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:16,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:20,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:22,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:22,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:53:22,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:53:28,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:30,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:32,604 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:53:33,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:37,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:53:37,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:53:39,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:40,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:45,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:50,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:53:51,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:53:52,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:53:52,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:53:57,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:00,083 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:54:00,090 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:54:00,090 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:54:00,090 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:54:00,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:00,965 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:54:02,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:02,607 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:54:02,966 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:54:07,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:54:07,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:54:07,525 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:10,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:13,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:19,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:20,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:22,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:54:22,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:54:25,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:30,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:30,666 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:32,608 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:54:35,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:37,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:54:37,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:54:40,667 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:41,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:46,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:50,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:54:51,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:54:52,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:54:52,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:54:56,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:00,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:02,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:02,611 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:55:07,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:55:07,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:55:07,482 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:10,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:13,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:18,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:20,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:22,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:55:22,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:55:23,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:29,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:30,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:32,614 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:55:34,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:37,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:55:37,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:55:40,124 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:40,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:45,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:50,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:55:51,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:55:52,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:55:52,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:55:57,125 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:00,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:01,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:56:01,852 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:56:01,852 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:56:01,853 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:56:02,040 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:56:02,616 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:56:02,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:03,041 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:56:07,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:56:07,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:56:08,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:10,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:14,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:19,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:20,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:22,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:56:22,332 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:56:25,133 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:30,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:31,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:32,618 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:56:36,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:37,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:56:37,333 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:56:40,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:41,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:47,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:50,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:56:53,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:56:53,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:56:53,257 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:56:59,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:00,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:02,620 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:57:04,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:08,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:57:08,257 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:57:10,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:10,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:15,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:20,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:21,135 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:23,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:57:23,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:57:26,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:30,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:32,136 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:32,623 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:57:37,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:38,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:57:38,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:57:40,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:43,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:49,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:57:50,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:57:53,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:57:53,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:57:55,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:00,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:00,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:02,625 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:58:04,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 20:58:04,691 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 20:58:04,691 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 20:58:04,692 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 20:58:05,107 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 20:58:05,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:07,109 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 20:58:08,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:58:08,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:58:10,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:11,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:17,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:20,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:23,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:23,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:58:23,258 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:58:29,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:30,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:32,631 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:58:34,138 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:38,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:58:38,259 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:58:40,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:40,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:45,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:50,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:58:50,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:58:53,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:58:53,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:58:56,139 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:00,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:01,140 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:02,630 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:59:06,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:08,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:59:08,259 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:59:10,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:12,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:17,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:20,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:23,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:23,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:59:23,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:59:29,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:30,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:32,634 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 20:59:34,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:38,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:59:38,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:59:40,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:40,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:46,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:50,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 20:59:52,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 20:59:53,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 20:59:53,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 20:59:58,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:00,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:02,635 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:00:03,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:06,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:00:06,227 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:00:06,228 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:00:06,228 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:00:07,172 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:00:08,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:00:08,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:00:09,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:09,173 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:00:10,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:15,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:20,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:20,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:23,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:00:23,260 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:00:26,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:30,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:32,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:32,637 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:00:37,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:38,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:00:38,261 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:00:40,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:43,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:48,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:50,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:00:53,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:00:53,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:00:53,261 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:00:58,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:00,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:02,640 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:01:03,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:08,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:01:08,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:01:09,146 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:10,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:14,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:19,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:20,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:23,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:01:23,261 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:01:25,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:30,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:31,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:32,642 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:01:36,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:38,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:01:38,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:01:40,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:42,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:47,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:50,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:01:53,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:01:53,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:01:53,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:01:59,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:00,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:02,645 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:02:05,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:07,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:02:07,972 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:02:07,972 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:02:07,973 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:02:08,220 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:02:08,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:02:08,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:02:09,221 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:02:10,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:10,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:16,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:20,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:22,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:23,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:02:23,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:02:27,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:30,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:32,647 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:02:32,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:38,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:38,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:02:38,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:02:40,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:44,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:50,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:02:50,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:02:53,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:02:53,262 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:02:55,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:00,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:00,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:02,652 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:03:06,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:08,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:03:08,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:03:10,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:12,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:18,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:20,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:23,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:23,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:03:23,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:03:29,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:30,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:32,651 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:03:35,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:38,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:03:38,263 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:03:40,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:41,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:47,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:50,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:03:52,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:03:53,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:03:53,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:03:57,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:00,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:02,655 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:04:02,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:08,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:08,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:04:08,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:04:10,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:11,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:04:11,037 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:04:11,037 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:04:11,038 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:04:11,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:04:13,270 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:04:14,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:19,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:20,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:23,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:04:23,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:04:25,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:30,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:30,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:32,656 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:04:35,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:38,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:04:38,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:04:40,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:40,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:45,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:50,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:04:51,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:04:53,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:04:53,264 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:04:57,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:00,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:02,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:02,659 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:05:07,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:08,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:05:08,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:05:10,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:13,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:19,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:20,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:23,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:05:23,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:05:25,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:30,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:31,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:32,661 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:05:36,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:38,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:05:38,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:05:40,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:42,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:48,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:50,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:05:53,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:05:53,282 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:05:53,494 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:05:59,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:00,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:02,664 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:06:04,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:08,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:06:08,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:06:09,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:10,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:12,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:06:12,732 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:06:12,732 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:06:12,733 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:06:13,324 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:06:14,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:15,325 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:06:20,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:20,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:23,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:06:23,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:06:25,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:30,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:31,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:32,666 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:06:36,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:38,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:06:38,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:06:40,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:42,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:48,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:50,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:06:53,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:06:53,283 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:06:53,438 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:06:59,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:00,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:02,668 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:07:05,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:08,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:07:08,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:07:10,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:11,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:16,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:20,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:22,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:23,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:07:23,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:07:28,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:30,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:32,674 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:07:33,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:38,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:07:38,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:07:39,182 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:40,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:44,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:50,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:07:50,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:07:53,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:07:53,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:07:56,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:00,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:01,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:02,673 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:08:06,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:08,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:08:08,284 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:08:10,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:12,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:14,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:08:14,294 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:08:14,294 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:08:14,296 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:08:14,381 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:08:17,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:17,382 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:08:20,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:23,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:23,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:08:23,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:08:28,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:30,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:32,675 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:08:33,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:38,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:08:38,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:08:39,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:40,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:44,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:49,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:08:50,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:08:53,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:08:53,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:08:55,197 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:00,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:00,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:02,680 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:09:05,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:08,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:09:08,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:09:10,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:11,198 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:16,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:20,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:22,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:23,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:09:23,286 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:09:28,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:30,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:32,680 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:09:33,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:38,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:09:38,286 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:09:39,200 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:40,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:44,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:50,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:09:50,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:09:53,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:09:53,286 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:09:56,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:00,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:02,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:02,682 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:10:07,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:08,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:10:08,286 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:10:10,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:13,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:17,481 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:10:17,482 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:10:17,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:10:17,483 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:10:18,434 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:10:18,523 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:19,434 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:10:20,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:23,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:10:23,286 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:10:24,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:29,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:30,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:32,685 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:10:34,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:38,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:10:38,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:10:40,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:40,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:46,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:50,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:10:52,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:10:53,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:10:53,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:10:57,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:00,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:02,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:02,689 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:11:07,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:08,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:11:08,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:11:10,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:13,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:19,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:20,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:23,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:11:23,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:11:24,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:29,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:30,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:32,751 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:11:35,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:38,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:11:38,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:11:40,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:40,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:45,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:50,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:11:50,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:11:53,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:11:53,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:11:56,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:00,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:01,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:02,757 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:12:06,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:08,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:12:08,287 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:12:10,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:12,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:18,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:19,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:12:19,116 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:12:19,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:12:19,117 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:12:19,477 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:12:20,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:21,478 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:12:23,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:12:23,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:12:23,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:29,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:30,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:32,756 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:12:34,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:38,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:12:38,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:12:40,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:40,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:45,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:50,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:12:51,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:12:53,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:12:53,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:12:57,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:00,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:02,759 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:13:02,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:08,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:08,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:13:08,288 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:13:10,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:14,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:19,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:20,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:23,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:13:23,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:13:24,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:30,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:30,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:32,760 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:13:35,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:38,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:13:38,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:13:40,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:41,246 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:46,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:50,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:13:52,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:13:53,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:13:53,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:13:57,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:00,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:02,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:02,763 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:14:07,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:08,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:14:08,289 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:14:10,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:13,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:19,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:20,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:14:20,605 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:14:20,606 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:14:20,606 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:14:20,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:21,523 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:14:23,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:14:23,290 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:14:23,524 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:14:25,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:30,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:31,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:32,765 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:14:36,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:38,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:14:38,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:14:40,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:42,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:48,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:50,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:14:53,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:14:53,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:14:53,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:14:59,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:00,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:02,767 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:15:04,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:08,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:15:08,295 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:15:10,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:10,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:15,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:20,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:21,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:23,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:15:23,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:15:27,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:30,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:32,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:32,770 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:15:37,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:38,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:15:38,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:15:40,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:42,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:48,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:50,752 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:15:53,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:15:53,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:15:53,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:15:59,280 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:00,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:02,772 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:16:04,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:08,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:16:08,296 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:16:10,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:10,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:15,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:20,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:20,753 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:23,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:16:23,668 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:16:24,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:16:24,833 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:16:24,833 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:16:24,834 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:16:25,573 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:16:25,875 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:27,574 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:16:30,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:31,281 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:32,778 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:16:36,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:38,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:16:38,668 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:16:40,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:42,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:47,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:50,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:16:52,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:16:53,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:16:53,668 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:16:58,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:00,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:02,777 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:17:03,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:08,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:17:08,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:17:08,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:10,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:14,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:20,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:20,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:23,668 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:17:23,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:17:26,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:30,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:31,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:32,781 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:17:37,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:38,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:17:38,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:17:40,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:42,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:48,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:50,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:17:53,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:17:53,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:17:53,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:17:59,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:00,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:02,782 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:18:04,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:08,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:18:08,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:18:09,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:10,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:15,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:20,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:21,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:23,669 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:18:23,669 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:18:26,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:18:26,218 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:18:26,218 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:18:26,219 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:18:26,621 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:18:27,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:27,621 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:18:30,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:32,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:32,784 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:18:37,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:38,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:18:38,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:18:40,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:43,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:49,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:50,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:18:53,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:18:53,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:18:54,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:18:59,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:00,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:02,787 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:19:04,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:08,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:19:08,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:19:09,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:10,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:14,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:19,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:20,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:23,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:19:23,670 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:19:25,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:30,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:30,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:32,789 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:19:35,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:38,670 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:19:38,671 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:19:40,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:41,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:46,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:50,761 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:19:51,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:19:53,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:19:53,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:19:56,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:00,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:02,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:02,792 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:20:07,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:08,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:20:08,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:20:10,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:13,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:18,309 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:20,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:23,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:20:23,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:20:23,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:27,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:20:27,686 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:20:27,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:20:27,687 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:20:28,679 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:20:29,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:29,680 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:20:30,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:32,794 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:20:34,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:38,671 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:20:38,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:20:39,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:40,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:45,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:50,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:20:51,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:20:53,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:20:53,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:20:57,308 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:00,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:02,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:02,796 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:21:07,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:08,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:21:08,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:21:10,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:13,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:18,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:20,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:23,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:21:23,672 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:21:23,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:29,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:30,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:32,802 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:21:34,314 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:38,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:21:38,673 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:21:40,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:40,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:46,315 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:50,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:21:51,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:21:53,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:21:53,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:21:56,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:00,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:01,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:02,801 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:22:07,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:08,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:22:08,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:22:10,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:13,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:18,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:20,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:23,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:23,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:22:23,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:22:28,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:29,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:22:29,532 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:22:29,533 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:22:29,535 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:22:29,750 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:22:30,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:31,751 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:22:32,803 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:22:33,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:38,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:22:38,674 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:22:38,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:40,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:44,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:49,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:22:50,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:22:53,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:22:53,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:22:55,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:00,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:00,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:02,807 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:23:06,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:08,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:23:08,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:23:10,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:11,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:16,338 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:20,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:21,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:23,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:23:23,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:23:27,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:30,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:32,340 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:32,808 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:23:37,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:38,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:23:38,675 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:23:40,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:43,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:49,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:23:50,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:23:53,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:23:53,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:23:55,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:00,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:00,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:02,810 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:24:05,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:08,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:24:08,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:24:10,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:11,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:16,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:20,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:22,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:23,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:24:23,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:24:28,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:30,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:32,812 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:24:33,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:24:33,117 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:24:33,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:24:33,119 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:24:33,822 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:24:34,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:35,823 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:24:38,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:24:38,676 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:24:39,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:40,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:44,348 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:50,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:24:50,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:24:53,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:24:53,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:24:56,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:00,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:02,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:02,815 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:25:07,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:08,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:25:08,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:25:10,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:13,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:19,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:20,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:23,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:25:23,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:25:25,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:30,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:31,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:32,815 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:25:37,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:38,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:25:38,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:25:40,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:42,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:48,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:50,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:25:53,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:25:53,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:25:53,677 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:25:59,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:00,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:02,820 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:26:04,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:08,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:26:08,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:26:09,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:10,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:15,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:20,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:20,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:23,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:26:23,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:26:25,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:30,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:31,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:32,824 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:26:35,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:26:35,122 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:26:35,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:26:35,124 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:26:35,896 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:26:37,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:37,895 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:26:38,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:26:38,678 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:26:40,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:42,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:48,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:50,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:26:53,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:26:53,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:26:53,896 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:26:59,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:00,772 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:02,823 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:27:04,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:08,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:27:08,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:27:09,356 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:10,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:15,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:20,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:21,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:23,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:27:23,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:27:26,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:30,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:32,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:32,827 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:27:37,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:38,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:27:38,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:27:40,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:42,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:47,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:50,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:27:53,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:27:53,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:27:53,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:27:59,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:00,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:02,828 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:28:05,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:08,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:28:08,679 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:28:10,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:10,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:16,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:20,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:22,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:23,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:28:23,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:28:28,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:30,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:32,830 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:28:33,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:37,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:28:37,130 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:28:37,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:28:37,131 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:28:37,964 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:28:38,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:28:38,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:28:39,091 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:39,965 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:28:40,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:44,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:49,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:28:50,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:28:53,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:28:53,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:28:55,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:00,363 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:00,775 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:02,833 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:29:06,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:08,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:29:08,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:29:10,776 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:11,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:17,364 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:20,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:23,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:23,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:29:23,680 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:29:29,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:30,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:32,835 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:29:34,836 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:38,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:29:38,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:29:39,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:40,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:45,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:50,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:29:50,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:29:53,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:29:53,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:29:55,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:00,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:00,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:02,837 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:30:05,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:08,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:30:08,681 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:30:10,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:11,253 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:17,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:20,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:23,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:23,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:30:23,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:30:28,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:30,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:32,839 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:30:33,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:38,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:30:38,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:30:38,941 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:40,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:30:40,397 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:30:40,398 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:30:40,398 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:30:40,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:41,039 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:30:42,039 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:30:44,440 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:50,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:30:50,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:30:53,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:30:53,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:30:55,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:00,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:00,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:02,890 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:31:05,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:08,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:31:08,682 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:31:10,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:10,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:16,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:20,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:21,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:23,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:31:23,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:31:27,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:30,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:32,845 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:31:32,847 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:38,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:38,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:31:38,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:31:40,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:44,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:49,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:31:50,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:31:53,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:31:53,683 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:31:54,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:00,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:00,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:02,848 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:32:05,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:08,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:32:08,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:32:10,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:11,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:16,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:20,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:22,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:23,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:32:23,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:32:28,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:30,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:32,850 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:32:33,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:38,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:32:38,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:32:38,944 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:40,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:44,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:49,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:32:50,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:32:53,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:32:53,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:32:54,918 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:00,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:00,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:02,853 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:33:03,172 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:33:03,173 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:33:03,173 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:33:03,174 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:33:04,112 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:33:05,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:06,113 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:33:08,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:33:08,684 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:33:10,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:10,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:16,263 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:20,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:21,276 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:23,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:33:23,685 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:33:27,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:30,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:32,856 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:33:32,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:38,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:38,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:33:38,685 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:33:40,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:44,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:49,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:33:50,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:33:53,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:33:53,685 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:33:54,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:00,266 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:00,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:02,859 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:34:05,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:08,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:34:08,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:34:10,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:11,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:16,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:20,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:22,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:23,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:34:23,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:34:27,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:30,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:32,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:32,861 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:34:38,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:38,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:34:38,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:34:40,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:44,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:49,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:34:50,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:34:53,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:34:53,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:34:54,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:00,284 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:00,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:02,864 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:35:04,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:35:04,723 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:35:04,723 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:35:04,724 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:35:05,171 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:35:06,172 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:35:06,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:08,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:35:08,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:35:10,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:12,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:17,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:20,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:23,285 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:23,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:35:23,686 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:35:28,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:30,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:32,870 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:35:33,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:38,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:35:38,687 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:35:38,934 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:40,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:44,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:49,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:35:50,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:35:53,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:35:53,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:35:54,899 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:00,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:00,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:02,869 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:36:05,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:08,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:36:08,688 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:36:10,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:11,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:17,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:20,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:23,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:23,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:36:23,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:36:28,289 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:30,805 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:32,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:36:33,876 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:38,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:36:38,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:36:38,933 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:40,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:44,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:50,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:36:50,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:36:53,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:36:53,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:36:55,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:00,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:00,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:02,873 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:37:05,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:08,128 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:37:08,130 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:37:08,130 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:37:08,131 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:37:08,244 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:37:08,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:37:08,689 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:37:09,244 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:37:10,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:11,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:16,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:20,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:21,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:23,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:37:23,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:37:27,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:30,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:32,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:32,875 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:37:38,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:38,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:37:38,690 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:37:40,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:43,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:48,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:50,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:37:53,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:37:53,694 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:37:53,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:37:59,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:00,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:02,879 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:38:04,880 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:08,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:38:08,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:38:10,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:10,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:16,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:20,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:21,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:23,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:38:23,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:38:27,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:30,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:32,881 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:38:32,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:38,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:38,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:38:38,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:38:40,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:43,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:49,298 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:38:50,808 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:38:53,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:38:53,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:38:54,889 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:00,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:00,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:02,884 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:39:05,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:08,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:39:08,695 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:39:09,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:39:09,860 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:39:09,860 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:39:09,861 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:39:10,309 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:39:10,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:11,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:11,309 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:39:16,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:20,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:22,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:23,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:39:23,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:39:28,301 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:30,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:32,886 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:39:33,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:38,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:39:38,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:39:38,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:40,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:44,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:49,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:39:50,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:39:53,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:39:53,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:39:54,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:00,302 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:00,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:02,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:40:05,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:08,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:40:08,696 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:40:10,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:11,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:16,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:20,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:22,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:23,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:40:23,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:40:27,304 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:30,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:32,325 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:32,891 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:40:37,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:38,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:40:38,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:40:40,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:43,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:48,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:50,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:40:53,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:40:53,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:40:53,950 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:40:59,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:00,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:02,895 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:41:04,898 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:08,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:41:08,697 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:41:10,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:10,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:11,700 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:41:11,701 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:41:11,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:41:11,703 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:41:12,375 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:41:13,376 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:41:15,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:20,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:20,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:23,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:41:23,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:41:26,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:30,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:31,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:32,896 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:41:37,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:38,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:41:38,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:41:40,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:42,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:48,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:50,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:41:53,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:41:53,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:41:54,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:41:59,334 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:00,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:02,898 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:42:04,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:08,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:42:08,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:42:09,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:10,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:15,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:20,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:20,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:23,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:42:23,698 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:42:26,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:30,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:31,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:32,902 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:42:36,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:38,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:42:38,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:42:40,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:42,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:47,350 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:50,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:42:53,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:42:53,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:42:53,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:42:59,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:00,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:02,904 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:43:04,906 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:08,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:43:08,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:43:09,939 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:10,826 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:15,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:43:15,118 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:43:15,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:43:15,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:15,120 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:43:15,438 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:43:17,439 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:43:20,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:20,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:23,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:43:23,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:43:25,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:30,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:31,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:32,906 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:43:37,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:38,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:43:38,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:43:40,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:43,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:48,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:50,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:43:53,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:43:53,699 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:43:54,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:43:59,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:00,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:02,909 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:44:04,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:08,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:44:08,702 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:44:09,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:10,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:15,354 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:20,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:20,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:23,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:44:23,710 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:44:25,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:30,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:31,375 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:32,912 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:44:36,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:38,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:44:38,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:44:40,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:42,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:48,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:50,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:44:53,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:44:53,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:44:53,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:44:59,376 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:00,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:02,918 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:45:04,919 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:08,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:45:08,703 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:45:10,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:10,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:15,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:19,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:45:19,434 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:45:19,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:45:19,435 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:45:19,505 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:45:20,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:21,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:21,506 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:45:23,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:45:23,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:45:27,377 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:30,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:32,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:32,916 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:45:37,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:38,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:45:38,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:45:40,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:43,378 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:49,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:45:50,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:45:53,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:45:53,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:45:54,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:00,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:00,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:02,920 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:46:05,379 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:08,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:46:08,704 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:46:10,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:10,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:16,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:20,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:22,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:23,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:46:23,705 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:46:28,380 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:30,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:32,921 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:46:33,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:38,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:46:38,707 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:46:39,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:40,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:44,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:49,384 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:46:50,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:46:53,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:46:53,708 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:46:54,952 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:00,381 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:00,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:02,923 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:47:05,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:08,708 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:47:08,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:47:10,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:11,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:17,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:20,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:22,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:23,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:47:23,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:47:27,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:29,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:47:29,034 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:47:29,034 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:47:29,035 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:47:29,576 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:47:29,576 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:47:30,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:32,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:32,926 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:47:37,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:38,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:47:38,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:47:40,841 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:43,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:48,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:50,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:47:53,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:47:53,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:47:53,709 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:47:59,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:00,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:02,929 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:48:04,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:08,709 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:48:08,710 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:48:10,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:10,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:15,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:20,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:21,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:23,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:48:23,713 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:48:27,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:30,844 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:32,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:32,932 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:48:38,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:38,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:48:38,711 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:48:40,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:43,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:49,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:48:50,848 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:48:53,710 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:48:53,712 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:48:54,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:00,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:00,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:02,937 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:49:06,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:08,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:49:08,712 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:49:10,849 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:11,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:17,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:20,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:22,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:23,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:49:23,715 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:49:28,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:30,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:32,942 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:49:33,512 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:49:33,514 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:49:33,514 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:49:33,514 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:33,515 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:49:33,672 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:49:35,673 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:49:38,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:49:38,713 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:49:38,900 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:40,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:44,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:50,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:49:50,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:49:53,712 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:49:53,713 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:49:56,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:00,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:01,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:02,939 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:50:07,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:08,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:50:08,713 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:50:10,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:13,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:18,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:20,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:23,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:50:23,713 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:50:23,893 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:29,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:30,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:32,943 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:50:34,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:38,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:50:38,714 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:50:40,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:40,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:46,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:50,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:50:52,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:50:53,713 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:50:53,714 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:50:58,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:00,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:02,944 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:51:03,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:08,714 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:51:08,714 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:51:09,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:10,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:15,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:20,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:20,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:23,732 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:51:23,733 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:51:26,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:30,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:31,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:32,947 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:51:35,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:51:35,197 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:51:35,198 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:51:35,198 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:51:35,718 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:51:37,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:37,719 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:51:38,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:51:38,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:51:40,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:42,413 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:47,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:50,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:51:52,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:51:53,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:51:53,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:51:58,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:00,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:02,950 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:52:03,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:08,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:52:08,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:52:09,894 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:10,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:15,415 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:20,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:20,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:23,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:52:23,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:52:25,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:30,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:31,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:32,952 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:52:36,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:38,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:52:38,734 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:52:40,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:42,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:48,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:50,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:52:53,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:52:53,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:52:53,937 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:52:59,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:00,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:02,954 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:53:04,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:08,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:53:08,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:53:10,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:10,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:15,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:20,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:21,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:23,734 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:53:23,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:53:26,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:30,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:31,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:32,956 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:53:36,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:53:36,790 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:53:36,790 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:53:36,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:36,791 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:53:37,764 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:53:37,764 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:53:38,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:53:38,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:53:40,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:42,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:48,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:50,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:53:53,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:53:53,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:53:53,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:53:59,420 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:00,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:02,963 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:54:04,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:08,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:54:08,735 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:54:10,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:10,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:15,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:20,424 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:20,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:23,735 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:54:23,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:54:25,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:30,429 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:30,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:32,961 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:54:35,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:38,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:54:38,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:54:40,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:41,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:46,430 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:50,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:54:51,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:54:53,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:54:53,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:54:56,431 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:00,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:02,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:02,966 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:55:07,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:08,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:55:08,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:55:10,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:12,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:18,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:20,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:23,736 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:55:23,736 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:55:23,936 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:29,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:30,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:32,966 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:55:34,967 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:38,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:55:38,314 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:55:38,315 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:55:38,315 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:55:38,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:55:38,737 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:55:38,809 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:55:39,810 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:55:40,433 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:40,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:45,434 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:50,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:55:51,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:55:53,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:55:53,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:55:55,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:01,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:01,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:02,969 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:56:06,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:08,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:56:08,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:56:11,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:12,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:17,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:21,028 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:23,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:23,737 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:56:23,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:56:28,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:31,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:32,972 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:56:33,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:38,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:56:38,738 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:56:39,930 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:41,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:45,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:50,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:56:51,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:56:53,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:56:53,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:56:55,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:01,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:01,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:02,974 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:57:06,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:08,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:57:08,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:57:11,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:12,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:17,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:21,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:23,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:23,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:57:23,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:57:28,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:31,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:32,977 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:57:33,978 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:38,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:57:38,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:57:39,945 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:41,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:41,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:57:41,331 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:57:41,331 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:57:41,332 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:57:41,872 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:57:43,874 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:57:45,372 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:51,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:57:51,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:57:53,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:57:53,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:57:57,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:01,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:02,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:02,979 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:58:08,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:08,738 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:58:08,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:58:11,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:14,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:19,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:21,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:23,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:58:23,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:58:24,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:30,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:31,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:32,986 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:58:35,988 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:38,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:58:38,739 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:58:41,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:41,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:47,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:51,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:58:53,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:58:53,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:58:53,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:58:58,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:01,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:02,984 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:59:03,986 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:08,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:59:08,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:59:09,947 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:11,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:15,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:20,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:21,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:23,739 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:59:23,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:59:25,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:31,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:31,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:32,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 21:59:36,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:38,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:59:38,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:59:41,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:42,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:42,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 21:59:42,855 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 21:59:42,856 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 21:59:42,857 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 21:59:42,922 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 21:59:43,923 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 21:59:48,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:51,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 21:59:53,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 21:59:53,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 21:59:53,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 21:59:59,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:01,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:02,989 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:00:04,991 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:08,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:00:08,740 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:00:10,912 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:11,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:16,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:21,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:22,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:23,740 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:00:23,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:00:27,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:31,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:32,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:32,992 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:00:37,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:38,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:00:38,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:00:41,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:43,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:48,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:51,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:00:53,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:00:53,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:00:53,954 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:00:59,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:01,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:02,995 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:01:04,997 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:08,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:01:08,741 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:01:10,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:11,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:16,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:21,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:21,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:23,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:01:23,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:01:26,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:31,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:31,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:32,997 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:01:36,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:38,741 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:01:38,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:01:41,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:42,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:44,492 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:01:44,494 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:01:44,495 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:01:44,496 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:01:44,973 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:01:45,974 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:01:47,054 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:51,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:01:53,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:01:53,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:01:53,742 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:01:59,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:01,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:02,999 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:02:05,001 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:08,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:02:08,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:02:10,903 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:11,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:16,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:21,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:22,055 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:23,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:02:23,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:02:27,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:31,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:33,002 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:02:33,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:38,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:38,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:02:38,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:02:41,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:44,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:49,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:02:51,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:02:53,742 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:02:53,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:02:54,891 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:00,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:01,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:03,008 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:03:06,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:08,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:03:08,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:03:11,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:11,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:16,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:21,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:22,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:23,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:03:23,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:03:27,058 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:31,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:33,007 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:03:33,010 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:38,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:38,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:03:38,743 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:03:41,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:44,059 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:47,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:03:47,663 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:03:47,663 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:03:47,664 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:03:48,048 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:03:49,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:03:50,050 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:03:51,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:03:53,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:03:53,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:03:54,957 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:00,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:01,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:03,010 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:04:06,012 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:08,743 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:04:08,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:04:11,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:11,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:16,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:21,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:22,060 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:23,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:04:23,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:04:27,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:31,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:33,012 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:04:33,015 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:38,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:38,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:04:38,744 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:04:41,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:44,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:49,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:04:51,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:04:53,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:04:53,745 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:04:54,932 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:00,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:01,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:03,015 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:05:06,016 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:08,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:05:08,745 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:05:11,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:11,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:16,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:21,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:22,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:23,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:05:23,745 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:05:28,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:31,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:33,017 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:05:34,018 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:38,744 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:05:38,745 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:05:39,951 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:41,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:45,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:49,134 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:05:49,136 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:05:49,136 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:05:49,137 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:05:50,108 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:05:50,108 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:05:50,179 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:05:51,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:05:53,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:05:53,746 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:05:55,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:01,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:01,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:03,019 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:06:07,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:08,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:06:08,746 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:06:11,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:12,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:18,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:21,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:23,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:23,745 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:06:23,746 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:06:29,066 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:31,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:33,022 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:06:35,023 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:38,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:06:38,746 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:06:40,886 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:41,086 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:46,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:51,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:06:52,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:06:53,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:06:53,746 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:06:58,067 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:01,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:03,024 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:07:04,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:08,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:07:08,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:07:09,964 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:11,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:15,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:21,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:21,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:23,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:07:23,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:07:27,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:31,087 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:32,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:33,027 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:07:38,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:38,746 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:07:38,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:07:41,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:44,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:49,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:07:50,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:07:50,625 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:07:50,625 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:07:50,626 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:07:51,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:07:51,176 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:07:52,177 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:07:53,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:07:53,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:07:54,956 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:00,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:01,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:03,030 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:08:06,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:08,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:08:08,747 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:08:11,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:11,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:16,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:21,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:21,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:23,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:08:23,748 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:08:26,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:31,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:32,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:33,032 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:08:37,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:38,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:08:38,748 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:08:41,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:43,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:48,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:51,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:08:53,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:08:53,747 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:08:53,748 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:08:59,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:01,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:03,035 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:09:05,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:08,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:09:08,749 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:09:10,917 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:11,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:16,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:21,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:21,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:23,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:09:23,749 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:09:26,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:31,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:32,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:33,037 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:09:37,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:38,748 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:09:38,750 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:09:41,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:42,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:48,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:51,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:09:53,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:09:53,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:09:53,749 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:09:53,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:09:53,947 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:09:53,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:09:53,949 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:09:54,244 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:09:56,246 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:09:59,397 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:01,396 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:03,040 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:10:05,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:08,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:10:08,750 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:10:10,968 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:11,400 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:16,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:21,399 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:21,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:23,749 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:10:23,750 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:10:27,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:31,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:33,043 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:10:33,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:38,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:38,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:10:38,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:10:41,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:43,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:49,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:10:51,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:10:53,750 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:10:53,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:10:54,922 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:00,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:01,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:03,046 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:11:06,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:08,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:11:08,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:11:11,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:11,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:16,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:21,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:22,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:23,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:11:23,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:11:27,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:31,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:33,049 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:11:33,052 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:38,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:38,751 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:11:38,751 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:11:41,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:44,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:50,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:51,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:11:53,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:11:53,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:11:55,529 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:11:55,532 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:11:55,532 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:11:55,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:11:55,534 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:11:56,296 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:11:56,297 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:12:01,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:01,405 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:03,051 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:12:06,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:08,754 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:12:08,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:12:11,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:12,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:17,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:21,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:23,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:23,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:12:23,755 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:12:28,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:31,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:33,056 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:12:34,057 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:38,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:12:38,757 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:12:39,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:41,403 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:45,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:50,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:12:51,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:12:53,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:12:53,756 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:12:55,943 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:01,404 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:01,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:03,055 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:13:06,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:08,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:13:08,757 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:13:11,406 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:11,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:17,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:21,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:22,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:23,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:13:23,759 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:13:27,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:31,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:32,411 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:33,059 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:13:37,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:38,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:13:38,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:13:41,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:43,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:48,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:51,407 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:13:53,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:13:53,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:13:54,008 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:13:57,272 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:13:57,273 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:13:57,274 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:13:57,274 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:13:57,356 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:13:58,358 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:13:59,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:01,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:03,060 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:14:05,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:08,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:14:08,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:14:11,011 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:11,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:16,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:21,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:21,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:23,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:14:23,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:14:27,418 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:31,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:33,063 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:14:33,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:38,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:38,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:14:38,758 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:14:41,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:44,419 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:49,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:14:51,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:14:53,760 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:14:53,763 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:14:54,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:00,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:01,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:03,065 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:15:06,068 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:08,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:15:08,759 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:15:11,410 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:11,421 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:16,422 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:21,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:21,580 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:23,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:15:23,760 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:15:26,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:31,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:32,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:33,067 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:15:38,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:38,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:15:38,767 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:15:41,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:43,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:49,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:15:51,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:15:53,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:15:53,764 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:15:55,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:00,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:01,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:16:01,230 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:16:01,230 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:16:01,231 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:16:01,431 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:16:01,581 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:02,433 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:16:03,070 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:16:06,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:08,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:16:08,764 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:16:11,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:11,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:17,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:21,582 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:23,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:23,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:16:23,765 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:16:28,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:31,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:33,072 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:16:34,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:38,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:16:38,764 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:16:39,915 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:41,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:45,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:50,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:16:51,583 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:16:53,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:16:53,764 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:16:55,921 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:01,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:01,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:03,081 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:17:06,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:08,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:17:08,765 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:17:11,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:12,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:17,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:21,584 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:23,591 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:23,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:17:23,765 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:17:28,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:31,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:33,077 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:17:34,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:38,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:17:38,765 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:17:39,985 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:41,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:45,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:50,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:17:51,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:17:53,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:17:53,765 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:17:56,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:01,585 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:01,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:03,081 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:18:03,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:18:03,768 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:18:03,768 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:18:03,769 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:18:04,503 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:18:04,504 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:18:06,623 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:08,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:18:08,766 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:18:11,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:11,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:16,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:21,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:22,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:23,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:18:23,766 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:18:27,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:31,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:33,083 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:18:33,085 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:38,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:38,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:18:38,767 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:18:41,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:44,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:50,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:18:51,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:18:53,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:18:53,767 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:18:55,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:01,587 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:01,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:03,085 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:19:06,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:08,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:19:08,767 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:19:11,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:12,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:17,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:21,588 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:23,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:23,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:19:23,767 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:19:29,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:31,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:33,087 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:19:35,088 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:38,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:19:38,768 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:19:40,971 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:41,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:46,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:51,589 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:19:52,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:19:53,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:19:53,768 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:19:57,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:01,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:02,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:03,089 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:20:06,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:20:06,055 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:20:06,055 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:20:06,056 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:20:06,577 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:20:08,578 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:20:08,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:08,768 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:20:08,768 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:20:11,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:14,630 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:20,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:21,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:23,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:20:23,782 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:20:25,966 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:31,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:31,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:33,092 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:20:36,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:38,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:20:38,774 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:20:41,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:42,631 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:47,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:51,592 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:20:53,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:20:53,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:20:53,774 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:20:58,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:01,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:03,095 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:21:04,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:08,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:21:08,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:21:09,994 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:11,593 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:15,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:21,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:21,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:23,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:21:23,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:21:26,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:31,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:32,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:33,102 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:21:37,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:38,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:21:38,775 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:21:41,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:43,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:48,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:51,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:21:54,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:21:54,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:21:54,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:21:59,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:01,596 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:03,102 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:22:05,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:09,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:22:09,041 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:22:09,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:22:09,835 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:22:09,835 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:22:09,839 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:22:10,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:22:10,654 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:22:10,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:11,601 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:16,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:21,603 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:22,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:24,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:22:24,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:22:27,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:31,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:32,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:33,102 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:22:37,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:39,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:22:39,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:22:41,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:43,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:49,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:22:51,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:22:54,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:22:54,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:22:55,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:00,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:01,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:03,106 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:23:06,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:09,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:23:09,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:23:11,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:11,604 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:16,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:21,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:22,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:24,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:23:24,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:23:28,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:31,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:33,106 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:23:34,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:39,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:23:39,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:23:39,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:41,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:44,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:49,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:23:51,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:23:54,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:23:54,043 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:23:55,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:00,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:01,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:03,109 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:24:06,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:09,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:24:09,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:24:11,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:11,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:24:11,541 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:24:11,542 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:24:11,543 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:24:11,605 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:11,736 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:24:12,737 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:24:16,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:21,606 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:22,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:24,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:24:24,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:24:27,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:31,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:33,111 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:24:33,114 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:38,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:39,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:24:39,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:24:41,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:44,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:49,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:24:51,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:24:54,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:24:54,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:24:55,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:00,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:01,607 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:03,113 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:25:06,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:09,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:25:09,044 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:25:11,196 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:11,608 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:16,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:21,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:22,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:24,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:25:24,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:25:27,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:31,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:33,115 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:25:33,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:38,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:39,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:25:39,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:25:41,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:43,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:49,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:25:51,611 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:25:54,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:25:54,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:25:55,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:00,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:01,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:03,121 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:26:06,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:09,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:26:09,045 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:26:11,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:11,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:13,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:26:13,256 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:26:13,256 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:26:13,257 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:26:13,817 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:26:14,818 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:26:16,299 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:21,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:21,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:24,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:26:24,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:26:26,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:31,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:32,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:33,120 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:26:37,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:39,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:26:39,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:26:41,612 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:43,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:48,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:51,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:26:54,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:26:54,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:26:54,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:26:59,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:01,613 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:03,125 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:27:05,127 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:09,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:27:09,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:27:10,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:11,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:15,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:21,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:21,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:24,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:27:24,047 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:27:26,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:31,614 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:32,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:33,126 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:27:37,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:39,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:27:39,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:27:41,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:43,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:48,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:51,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:27:54,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:27:54,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:27:54,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:27:59,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:01,615 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:03,129 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:28:05,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:09,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:28:09,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:28:10,268 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:11,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:15,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:28:15,092 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:28:15,092 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:28:15,093 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:28:15,922 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:28:16,131 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:16,923 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:28:21,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:21,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:24,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:28:24,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:28:26,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:31,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:31,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:33,132 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:28:37,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:39,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:28:39,048 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:28:41,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:43,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:48,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:51,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:28:54,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:28:54,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:28:54,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:28:59,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:01,616 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:03,135 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:29:05,137 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:09,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:29:09,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:29:10,255 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:11,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:15,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:20,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:21,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:24,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:29:24,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:29:26,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:31,618 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:31,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:33,137 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:29:36,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:39,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:29:39,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:29:41,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:42,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:47,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:51,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:29:53,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:29:54,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:29:54,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:29:58,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:01,619 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:03,139 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:30:04,141 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:09,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:30:09,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:30:09,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:11,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:14,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:18,617 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:30:18,619 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:30:18,619 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:30:18,620 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:30:19,016 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:30:20,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:21,018 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:30:21,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:24,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:30:24,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:30:26,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:31,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:31,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:33,142 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:30:36,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:39,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:30:39,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:30:41,620 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:42,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:47,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:51,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:30:53,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:30:54,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:30:54,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:30:58,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:01,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:03,148 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:31:04,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:09,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:31:09,061 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:31:09,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:11,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:14,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:20,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:21,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:24,061 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:31:24,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:31:26,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:31,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:31,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:33,147 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:31:36,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:39,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:31:39,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:31:41,624 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:42,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:47,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:51,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:31:53,690 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:31:54,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:31:54,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:31:58,691 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:01,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:03,151 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:32:04,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:09,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:32:09,062 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:32:09,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:11,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:14,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:19,693 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:20,459 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:32:20,461 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:32:20,462 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:32:20,462 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:32:21,100 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:32:21,625 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:23,101 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:32:24,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:32:24,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:32:25,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:30,692 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:31,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:33,152 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:32:36,154 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:39,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:32:39,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:32:41,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:41,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:46,694 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:51,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:32:51,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:32:54,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:32:54,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:32:56,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:01,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:02,695 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:03,154 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:33:07,696 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:09,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:33:09,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:33:11,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:12,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:18,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:21,626 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:23,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:24,062 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:33:24,063 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:33:29,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:31,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:33,157 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:33:35,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:39,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:33:39,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:33:40,261 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:41,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:45,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:50,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:33:51,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:33:54,063 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:33:54,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:33:56,195 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:01,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:01,702 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:03,159 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:34:06,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:09,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:34:09,064 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:34:11,627 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:12,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:17,703 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:21,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:21,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:34:21,992 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:34:21,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:34:21,993 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:34:22,154 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:34:23,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:23,155 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:34:24,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:34:24,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:34:28,704 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:31,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:33,162 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:34:34,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:39,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:34:39,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:34:39,287 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:41,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:44,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:49,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:34:51,628 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:34:54,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:34:54,065 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:34:55,234 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:00,705 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:01,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:03,164 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:35:06,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:09,064 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:35:09,066 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:35:11,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:11,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:16,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:21,629 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:21,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:24,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:35:24,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:35:26,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:31,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:32,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:33,170 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:35:37,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:39,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:35:39,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:35:41,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:43,706 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:48,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:51,633 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:35:54,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:35:54,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:35:54,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:35:59,707 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:01,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:03,169 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:36:05,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:09,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:36:09,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:36:10,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:11,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:15,720 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:20,721 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:21,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:24,069 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:36:24,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:36:24,959 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:36:24,961 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:36:24,961 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:36:24,964 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:36:25,212 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:36:25,965 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:27,213 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:36:31,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:31,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:33,171 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:36:36,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:39,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:36:39,070 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:36:41,634 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:42,722 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:47,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:51,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:36:53,723 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:36:54,070 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:36:54,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:36:58,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:01,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:03,175 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:37:04,176 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:09,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:37:09,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:37:09,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:11,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:14,724 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:19,725 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:21,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:24,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:37:24,071 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:37:25,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:30,726 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:31,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:33,176 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:37:36,178 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:39,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:37:39,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:37:41,297 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:41,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:46,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:51,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:37:52,730 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:37:54,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:37:54,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:37:57,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:01,636 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:03,178 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:38:03,181 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:08,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:09,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:38:09,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:38:11,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:14,731 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:19,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:21,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:24,071 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:38:24,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:38:25,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:26,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:38:26,482 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:38:26,482 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:38:26,483 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:38:27,269 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:38:29,270 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:38:30,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:31,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:33,181 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:38:36,183 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:39,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:38:39,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:38:41,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:41,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:46,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:51,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:38:51,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:38:54,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:38:54,072 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:38:57,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:01,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:02,733 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:03,184 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:39:07,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:09,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:39:09,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:39:11,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:13,755 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:18,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:21,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:24,072 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:39:24,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:39:24,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:29,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:31,638 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:33,187 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:39:35,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:39,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:39:39,073 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:39:40,313 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:41,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:45,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:51,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:39:51,756 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:39:54,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:39:54,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:39:56,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:01,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:02,757 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:03,192 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:40:07,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:09,073 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:40:09,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:40:11,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:13,758 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:18,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:21,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:24,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:40:24,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:40:24,269 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:28,432 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:40:28,434 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:40:28,434 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:40:28,435 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:40:29,337 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:40:29,475 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:31,338 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:40:31,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:33,192 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:40:35,193 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:39,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:40:39,074 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:40:40,236 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:41,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:45,759 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:50,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:40:51,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:40:54,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:40:54,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:40:56,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:01,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:01,762 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:03,193 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:41:06,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:09,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:41:09,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:41:11,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:11,763 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:16,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:21,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:21,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:24,074 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:41:24,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:41:26,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:31,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:32,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:33,196 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:41:37,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:39,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:41:39,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:41:41,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:43,765 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:48,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:51,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:41:53,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:41:54,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:41:54,075 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:41:59,767 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:01,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:03,200 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:42:05,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:09,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:42:09,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:42:10,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:11,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:15,766 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:20,769 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:21,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:24,075 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:42:24,076 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:42:25,774 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:31,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:31,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:31,962 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:42:31,964 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:42:31,964 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:42:31,965 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:42:32,400 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:42:32,400 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:42:33,201 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:42:36,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:39,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:42:39,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:42:41,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:42,770 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:47,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:51,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:42:52,777 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:42:54,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:42:54,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:42:57,778 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:01,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:03,206 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:43:03,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:08,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:09,076 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:43:09,077 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:43:11,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:13,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:18,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:21,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:24,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:43:24,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:43:24,271 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:29,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:31,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:33,207 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:43:35,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:39,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:43:39,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:43:40,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:41,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:45,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:51,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:43:51,780 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:43:54,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:43:54,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:43:56,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:01,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:01,781 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:03,209 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:44:06,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:09,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:44:09,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:44:11,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:12,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:17,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:21,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:23,782 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:24,077 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:44:24,078 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:44:28,783 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:31,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:33,217 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:44:33,970 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:44:33,973 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:44:33,973 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:44:33,974 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:33,974 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:44:34,459 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:44:34,459 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:44:39,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:44:39,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:44:39,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:41,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:44,785 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:49,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:44:51,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:44:54,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:44:54,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:44:55,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:00,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:01,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:03,214 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:45:06,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:09,078 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:45:09,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:45:11,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:11,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:16,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:21,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:22,786 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:24,079 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:45:24,079 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:45:27,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:31,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:33,218 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:45:33,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:38,787 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:39,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:45:39,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:45:41,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:44,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:50,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:45:51,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:45:54,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:45:54,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:45:56,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:01,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:01,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:03,220 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:46:06,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:09,080 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:46:09,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:46:11,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:12,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:18,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:21,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:24,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:46:24,082 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:46:24,546 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:29,790 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:31,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:33,222 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:46:35,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:38,990 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:46:38,992 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:46:38,992 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:46:38,993 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:46:39,081 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:46:39,081 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:46:39,519 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:46:40,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:40,520 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:46:41,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:45,791 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:50,793 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:46:51,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:46:54,082 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:46:54,083 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:46:56,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:01,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:01,792 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:03,225 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:47:06,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:09,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:47:09,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:47:11,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:12,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:18,794 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:21,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:24,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:47:24,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:47:24,352 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:29,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:31,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:33,227 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:47:35,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:39,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:47:39,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:47:40,341 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:41,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:45,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:51,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:47:51,796 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:47:54,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:47:54,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:47:57,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:01,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:03,230 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:48:03,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:08,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:09,095 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:48:09,095 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:48:11,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:14,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:20,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:21,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:24,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:48:24,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:48:26,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:31,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:31,797 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:33,232 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:48:36,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:39,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:48:39,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:48:41,652 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:42,541 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:48:42,542 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:48:42,543 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:48:42,543 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:48:42,544 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:42,577 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:48:44,579 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:48:47,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:51,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:48:53,798 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:48:54,096 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:48:54,096 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:48:58,799 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:01,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:03,239 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:49:04,241 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:09,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:49:09,097 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:49:09,343 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:11,653 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:14,800 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:19,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:21,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:24,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:49:24,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:49:25,357 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:30,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:31,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:33,237 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:49:36,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:39,097 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:49:39,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:49:41,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:41,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:46,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:51,654 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:49:51,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:49:54,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:49:54,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:49:56,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:01,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:02,802 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:03,242 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:50:07,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:09,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:50:09,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:50:11,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:13,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:18,804 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:21,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:23,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:24,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:50:24,098 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:50:29,809 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:31,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:33,242 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:50:35,243 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:39,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:50:39,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:50:40,471 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:41,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:45,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:48,499 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:50:48,501 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:50:48,501 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:50:48,502 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:50:48,646 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:50:50,647 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:50:51,543 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:50:51,655 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:50:54,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:50:54,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:50:56,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:01,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:02,810 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:03,245 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:51:07,827 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:09,098 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:51:09,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:51:11,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:13,811 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:18,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:21,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:23,812 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:24,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:51:24,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:51:28,813 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:31,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:33,248 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:51:34,249 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:39,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:51:39,099 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:51:39,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:41,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:44,814 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:50,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:51:51,656 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:51:54,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:51:54,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:51:56,353 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:01,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:01,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:03,252 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:52:06,815 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:09,099 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:52:09,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:52:11,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:11,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:17,816 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:21,657 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:23,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:24,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:52:24,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:52:28,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:31,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:33,253 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:52:34,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:39,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:52:39,100 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:52:39,484 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:41,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:44,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:49,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:52:50,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:52:50,153 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:52:50,153 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:52:50,154 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:52:50,706 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:52:50,706 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:52:51,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:52:54,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:52:54,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:52:55,349 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:00,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:01,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:03,256 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:53:06,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:09,100 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:53:09,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:53:11,463 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:11,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:16,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:21,658 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:22,819 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:24,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:53:24,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:53:28,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:31,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:33,258 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:53:34,260 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:39,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:53:39,101 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:53:39,361 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:41,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:44,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:49,821 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:53:51,659 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:53:54,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:53:54,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:53:55,312 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:00,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:01,660 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:03,265 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:54:06,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:09,101 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:54:09,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:54:11,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:11,661 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:16,822 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:21,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:22,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:24,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:54:24,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:54:27,823 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:31,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:33,263 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:54:33,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:38,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:39,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:54:39,102 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:54:41,662 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:43,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:48,824 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:51,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:54:53,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:54:53,253 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:54:53,254 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:54:53,256 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:54:53,769 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:54:54,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:54:54,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:54:54,358 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:54:54,770 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:54:59,825 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:01,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:03,266 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:55:05,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:09,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:55:09,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:55:11,247 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:11,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:16,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:21,663 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:21,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:24,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:55:24,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:55:26,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:31,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:31,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:33,270 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:55:36,829 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:39,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:55:39,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:55:41,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:42,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:48,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:51,664 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:55:53,838 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:55:54,102 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:55:54,103 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:55:59,831 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:01,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:03,271 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:56:05,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:09,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:56:09,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:56:10,355 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:11,665 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:15,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:20,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:21,672 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:24,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:56:24,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:56:26,288 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:31,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:31,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:33,273 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:56:37,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:39,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:56:39,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:56:41,673 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:43,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:48,832 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:51,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:56:54,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:56:54,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:56:54,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:56:55,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:56:55,120 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:56:55,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:56:55,121 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:56:55,841 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:56:56,842 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:56:59,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:01,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:03,275 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:57:05,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:09,103 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:57:09,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:57:11,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:11,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:16,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:21,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:22,833 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:24,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:57:24,104 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:57:27,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:31,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:32,834 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:33,277 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:57:37,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:39,104 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:57:39,105 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:57:41,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:43,835 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:48,839 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:51,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:57:54,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:57:54,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:57:54,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:57:59,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:01,675 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:03,280 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:58:05,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:09,106 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:58:09,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:58:11,401 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:11,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:16,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:21,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:22,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:24,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:58:24,107 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:58:28,850 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:31,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:33,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:58:34,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:39,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:58:39,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:58:39,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:41,676 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:44,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:50,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:51,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:58:54,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:58:54,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:58:56,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:58:56,863 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 22:58:56,864 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 22:58:56,865 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 22:58:56,866 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 22:58:56,901 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 22:58:58,902 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 22:59:01,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:01,851 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:03,284 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:59:06,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:09,107 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:59:09,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:59:11,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:12,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:17,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:21,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:23,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:24,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:59:24,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:59:28,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:31,677 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:33,288 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 22:59:34,290 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:39,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:59:39,108 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:59:39,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:41,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:44,853 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:49,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 22:59:51,678 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 22:59:54,108 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 22:59:54,109 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 22:59:55,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:00,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:01,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:03,290 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:00:06,291 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:09,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:00:09,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:00:11,516 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:11,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:16,855 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:21,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:21,858 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:24,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:00:24,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:00:27,856 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:31,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:33,292 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:00:33,293 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:38,857 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:39,109 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:00:39,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:00:41,679 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:43,862 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:48,867 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:51,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:00:53,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:00:54,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:00:54,110 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:00:58,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:01,680 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:01,949 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:01:01,951 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:01:01,952 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:01:01,952 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:01:01,976 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:01:02,976 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:01:03,294 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:01:04,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:09,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:01:09,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:01:10,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:11,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:15,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:21,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:21,869 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:24,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:01:24,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:01:26,870 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:31,681 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:32,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:33,300 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:01:38,871 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:39,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:01:39,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:01:41,682 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:43,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:49,872 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:01:51,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:01:54,110 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:01:54,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:01:55,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:00,877 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:01,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:03,302 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:02:06,306 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:09,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:02:09,111 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:02:11,336 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:11,683 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:16,873 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:21,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:21,874 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:24,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:02:24,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:02:26,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:31,684 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:32,878 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:33,304 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:02:37,879 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:39,111 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:02:39,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:02:41,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:42,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:48,882 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:51,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:02:54,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:02:54,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:02:54,322 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:02:59,881 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:01,685 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:03,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:03:04,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:03:04,915 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:03:04,916 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:03:04,916 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:04,917 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:03:05,061 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:03:07,062 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:03:09,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:03:09,112 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:03:10,326 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:11,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:15,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:21,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:21,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:24,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:03:24,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:03:27,883 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:31,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:32,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:33,310 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:03:37,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:39,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:03:39,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:03:41,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:43,884 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:48,885 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:51,686 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:03:54,112 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:03:54,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:03:54,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:03:59,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:01,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:03,314 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:04:05,316 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:09,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:04:09,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:04:11,303 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:11,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:16,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:21,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:21,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:24,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:04:24,113 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:04:26,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:31,687 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:32,887 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:33,315 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:04:37,888 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:39,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:04:39,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:04:41,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:42,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:48,904 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:51,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:04:53,905 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:04:54,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:04:54,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:04:58,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:01,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:03,326 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:05:04,330 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:06,946 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:05:06,947 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:05:06,948 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:05:06,949 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:05:07,115 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:05:09,113 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:05:09,114 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:05:09,116 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:05:10,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:11,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:15,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:20,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:21,688 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:24,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:05:24,125 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:05:26,264 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:31,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:31,909 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:33,322 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:05:36,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:39,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:05:39,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:05:41,689 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:42,910 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:48,911 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:51,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:05:54,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:05:54,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:05:54,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:05:59,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:01,697 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:03,327 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:06:05,331 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:09,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:06:09,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:06:11,292 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:11,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:16,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:21,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:22,913 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:24,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:06:24,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:06:27,914 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:31,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:32,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:33,327 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:06:37,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:39,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:06:39,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:06:41,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:43,924 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:48,925 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:51,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:06:54,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:06:54,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:06:54,283 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:06:59,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:01,698 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:03,329 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:07:05,333 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:08,902 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:07:08,904 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:07:08,904 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:07:08,905 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:07:09,115 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:07:09,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:07:09,184 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:07:11,185 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:07:11,267 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:11,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:16,926 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:21,927 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:23,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:24,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:07:24,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:07:27,265 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:33,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:33,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:33,335 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:07:38,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:39,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:07:39,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:07:43,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:43,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:49,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:07:53,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:07:54,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:07:54,116 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:07:54,254 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:00,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:03,215 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:03,334 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:08:05,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:09,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:08:09,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:08:11,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:13,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:17,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:22,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:23,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:24,116 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:08:24,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:08:27,337 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:33,216 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:33,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:33,338 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:08:38,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:39,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:08:39,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:08:43,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:44,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:49,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:08:53,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:08:54,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:08:54,117 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:08:54,274 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:00,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:03,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:03,339 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:09:05,342 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:09,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:09:09,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:09:11,256 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:12,065 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:09:12,068 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:09:12,068 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:09:12,069 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:09:12,241 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:09:13,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:13,241 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:09:17,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:22,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:23,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:24,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:09:24,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:09:27,310 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:33,219 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:33,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:33,341 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:09:38,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:39,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:09:39,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:09:43,221 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:44,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:49,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:09:53,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:09:54,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:09:54,118 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:09:54,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:00,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:03,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:03,343 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:10:05,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:09,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:10:09,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:10:11,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:13,222 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:17,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:22,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:23,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:24,118 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:10:24,119 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:10:27,307 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:33,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:33,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:33,346 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:10:38,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:39,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:10:39,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:10:43,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:43,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:48,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:10:53,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:10:54,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:10:54,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:10:54,296 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:00,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:03,224 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:03,348 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:11:05,351 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:09,119 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:11:09,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:11:11,295 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:13,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:13,586 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:11:13,588 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:11:13,588 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:11:13,589 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:11:14,295 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:11:15,296 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:11:16,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:22,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:23,225 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:24,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:11:24,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:11:27,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:33,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:33,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:33,351 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:11:38,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:39,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:11:39,120 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:11:43,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:44,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:49,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:11:53,226 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:11:54,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:11:54,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:11:54,273 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:00,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:03,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:03,356 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:12:05,360 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:09,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:12:09,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:12:11,279 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:13,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:17,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:22,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:23,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:24,120 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:12:24,121 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:12:27,252 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:33,227 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:33,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:33,355 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:12:38,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:39,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:12:39,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:12:43,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:44,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:49,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:12:53,228 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:12:54,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:12:54,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:12:54,277 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:00,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:03,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:03,360 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:13:05,362 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:09,121 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:13:09,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:13:11,294 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:13,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:15,436 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:13:15,437 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:13:15,438 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:13:15,439 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:13:16,353 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:13:16,479 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:17,354 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:13:22,235 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:23,229 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:24,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:13:24,122 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:13:27,275 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:33,230 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:33,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:33,361 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:13:39,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:13:39,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:13:39,305 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:43,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:45,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:50,238 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:13:53,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:13:54,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:13:54,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:13:55,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:01,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:03,231 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:03,363 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:14:06,365 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:09,122 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:14:09,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:14:12,318 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:13,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:18,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:23,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:24,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:14:24,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:14:24,339 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:30,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:33,232 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:33,365 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:14:35,368 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:39,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:14:39,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:14:41,270 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:43,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:47,240 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:52,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:14:53,233 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:14:54,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:14:54,124 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:14:57,346 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:03,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:03,257 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:03,368 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:15:08,259 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:09,123 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:15:09,123 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:15:13,237 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:14,258 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:18,590 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:15:18,591 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:15:18,592 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:15:18,593 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:15:19,405 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:15:19,635 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:21,406 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:15:24,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:26,327 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:26,328 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:15:26,328 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:15:29,409 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:15:30,524 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:33,373 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:15:36,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:36,344 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:37,418 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:15:41,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:15:41,329 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:15:41,519 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:46,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:47,395 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:53,394 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:15:56,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:15:56,392 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:15:56,402 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:15:58,594 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:03,398 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:16:04,402 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:06,393 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:10,398 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:11,329 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:16:11,330 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:16:15,533 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:16,542 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:20,555 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:26,335 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:16:26,336 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:16:26,478 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:26,530 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:31,521 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:33,404 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:16:36,537 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:37,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:41,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:16:41,601 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:16:42,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:46,540 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:47,214 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:52,217 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:16:56,538 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:16:56,600 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:16:56,600 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:16:57,218 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:02,220 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:03,404 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:17:06,539 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:07,223 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:11,699 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:17:11,700 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:17:12,332 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:16,568 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:18,262 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:23,278 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:26,632 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:17:26,633 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:17:26,674 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:28,323 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:33,324 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:33,407 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:17:36,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:39,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:41,637 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:17:41,638 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:17:44,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:46,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:49,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:54,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:17:56,595 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:17:56,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:17:56,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:17:59,089 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:03,409 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:18:04,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:06,597 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:10,117 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:11,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:18:11,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:18:15,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:16,621 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:20,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:26,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:26,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:26,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:18:26,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:18:31,142 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:33,411 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:18:36,143 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:36,610 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:41,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:18:41,639 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:18:41,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:47,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:48,025 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:53,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:18:56,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:18:56,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:18:58,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:18:58,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:03,414 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:19:04,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:08,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:19:10,144 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:11,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:19:11,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:19:15,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:18,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:19:21,145 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:26,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:26,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:19:26,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:19:28,026 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:19:32,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:33,416 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:19:38,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:19:38,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:41,639 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:19:41,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:19:43,803 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:48,029 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:19:49,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:55,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:19:56,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:19:56,640 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:19:58,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:00,147 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:03,418 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:20:05,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:08,030 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:11,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:11,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:20:11,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:20:17,148 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:18,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:22,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:26,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:20:26,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:20:27,846 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:28,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:33,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:33,420 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:20:38,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:38,149 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:41,640 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:20:41,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:20:43,807 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:48,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:20:49,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:54,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:20:56,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:20:56,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:20:58,031 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:00,150 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:03,423 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:21:05,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:08,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:10,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:11,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:21:11,641 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:21:16,151 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:18,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:22,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:26,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:21:26,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:21:27,830 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:28,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:33,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:33,430 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:21:38,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:38,152 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:41,641 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:21:41,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:21:43,854 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:48,032 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:21:49,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:54,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:21:56,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:21:56,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:21:58,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:00,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:03,429 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:22:06,153 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:08,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:11,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:22:11,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:22:11,773 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:17,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:18,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:22,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:26,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:22:26,642 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:22:27,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:28,033 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:33,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:33,432 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:22:38,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:38,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:41,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:22:41,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:22:43,820 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:48,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:49,155 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:54,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:22:56,642 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:22:56,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:22:58,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:22:59,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:03,433 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:23:05,156 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:08,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:23:11,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:11,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:23:11,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:23:16,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:18,034 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:23:22,157 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:26,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:23:26,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:23:27,837 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:28,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:23:33,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:33,435 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:23:38,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:23:39,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:41,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:23:41,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:23:44,158 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:48,035 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:23:49,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:55,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:23:56,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:23:56,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:23:58,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:01,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:03,438 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:24:07,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:08,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:11,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:24:11,643 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:24:12,840 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:18,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:18,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:24,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:26,643 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:24:26,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:24:28,036 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:30,159 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:33,441 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:24:35,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:38,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:41,160 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:41,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:24:41,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:24:46,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:48,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:24:52,161 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:56,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:24:56,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:24:57,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:24:58,037 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:03,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:03,443 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:25:08,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:08,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:11,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:25:11,644 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:25:13,764 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:18,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:19,162 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:24,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:26,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:25:26,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:25:28,038 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:30,163 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:33,445 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:25:35,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:38,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:41,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:41,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:25:41,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:25:46,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:48,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:25:52,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:56,644 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:25:56,645 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:25:57,771 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:25:58,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:03,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:03,456 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:26:08,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:09,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:11,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:26:11,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:26:14,164 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:18,039 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:20,165 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:25,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:26,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:26:26,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:26:28,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:31,166 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:33,450 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:26:36,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:38,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:41,645 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:26:41,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:26:41,864 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:47,167 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:48,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:53,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:26:56,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:26:56,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:26:58,040 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:26:58,859 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:03,454 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:27:04,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:08,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:27:10,168 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:11,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:27:11,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:27:16,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:18,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:27:22,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:26,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:27:26,646 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:27:27,843 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:28,041 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:27:33,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:33,455 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:27:38,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:27:38,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:41,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:27:41,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:27:43,865 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:48,042 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:27:49,169 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:55,170 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:27:56,646 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:27:56,647 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:27:58,046 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:00,173 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:03,457 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:28:05,184 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:08,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:11,185 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:11,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:28:11,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:28:16,186 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:18,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:21,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:26,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:28:26,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:28:26,788 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:28,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:32,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:33,459 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:28:38,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:38,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:41,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:28:41,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:28:43,818 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:48,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:28:49,187 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:54,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:28:56,647 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:28:56,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:28:58,043 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:00,188 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:03,462 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:29:05,189 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:08,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:10,190 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:11,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:29:11,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:29:15,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:18,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:21,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:26,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:29:26,648 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:29:26,806 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:28,044 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:32,191 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:33,465 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:29:37,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:38,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:41,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:29:41,651 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:29:42,779 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:48,045 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:48,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:53,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:29:56,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:29:56,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:29:58,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:29:58,795 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:03,468 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:30:04,199 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:08,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:09,203 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:11,648 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:30:11,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:30:15,201 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:18,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:21,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:26,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:26,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:30:26,649 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:30:28,048 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:32,202 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:33,470 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:30:37,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:38,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:41,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:30:41,650 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:30:42,852 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:48,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:48,204 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:53,212 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:30:56,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:30:56,650 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:30:58,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:30:58,817 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:03,477 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:31:04,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:08,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:31:10,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:11,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:31:11,650 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:31:16,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:18,049 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:31:21,205 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:26,649 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:31:26,650 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:31:26,789 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:28,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:31:32,206 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:33,475 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:31:37,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:38,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:31:41,650 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:31:41,651 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:31:42,282 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: partial_history 2024-08-31 23:31:42,284 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:31:42,285 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:31:42,286 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:42,288 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:31:42,957 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:31:43,958 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:31:48,207 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:50,977 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:31:53,963 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:31:53,982 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:31:56,651 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: stop_status 2024-08-31 23:31:56,652 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: stop_status 2024-08-31 23:31:59,842 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:32:00,796 DEBUG SenderThread:29126 [sender.py:send():391] send: telemetry 2024-08-31 23:32:00,801 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: check_version 2024-08-31 23:32:00,801 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: check_version 2024-08-31 23:32:00,828 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: server_info 2024-08-31 23:32:00,829 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: server_info 2024-08-31 23:32:01,002 DEBUG SenderThread:29126 [sender.py:send():391] send: exit 2024-08-31 23:32:01,002 INFO SenderThread:29126 [sender.py:send_exit():598] handling exit code: 0 2024-08-31 23:32:01,002 INFO SenderThread:29126 [sender.py:send_exit():600] handling runtime: 99156 2024-08-31 23:32:01,003 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:32:01,003 INFO SenderThread:29126 [sender.py:send_exit():606] send defer 2024-08-31 23:32:01,003 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,003 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 0 2024-08-31 23:32:01,004 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,004 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 0 2024-08-31 23:32:01,004 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 1 2024-08-31 23:32:01,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,004 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 1 2024-08-31 23:32:01,004 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,004 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 1 2024-08-31 23:32:01,004 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 2 2024-08-31 23:32:01,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,005 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 2 2024-08-31 23:32:01,005 INFO HandlerThread:29126 [system_monitor.py:finish():203] Stopping system monitor 2024-08-31 23:32:01,006 DEBUG SystemMonitor:29126 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-08-31 23:32:01,006 DEBUG SystemMonitor:29126 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-08-31 23:32:01,008 INFO HandlerThread:29126 [interfaces.py:finish():200] Joined cpu monitor 2024-08-31 23:32:01,009 INFO HandlerThread:29126 [interfaces.py:finish():200] Joined disk monitor 2024-08-31 23:32:01,044 INFO HandlerThread:29126 [interfaces.py:finish():200] Joined gpu monitor 2024-08-31 23:32:01,045 INFO HandlerThread:29126 [interfaces.py:finish():200] Joined memory monitor 2024-08-31 23:32:01,045 INFO HandlerThread:29126 [interfaces.py:finish():200] Joined network monitor 2024-08-31 23:32:01,046 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,046 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 2 2024-08-31 23:32:01,046 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 3 2024-08-31 23:32:01,046 DEBUG SenderThread:29126 [sender.py:send():391] send: stats 2024-08-31 23:32:01,047 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,047 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 3 2024-08-31 23:32:01,049 DEBUG SenderThread:29126 [sender.py:send():391] send: history 2024-08-31 23:32:01,049 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: summary_record 2024-08-31 23:32:01,049 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:32:01,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,050 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 3 2024-08-31 23:32:01,050 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 4 2024-08-31 23:32:01,050 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,050 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 4 2024-08-31 23:32:01,050 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,051 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 4 2024-08-31 23:32:01,051 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 5 2024-08-31 23:32:01,051 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,051 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 5 2024-08-31 23:32:01,052 DEBUG SenderThread:29126 [sender.py:send():391] send: summary 2024-08-31 23:32:01,052 INFO SenderThread:29126 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end 2024-08-31 23:32:01,052 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,052 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 5 2024-08-31 23:32:01,052 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 6 2024-08-31 23:32:01,053 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,053 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 6 2024-08-31 23:32:01,053 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,053 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 6 2024-08-31 23:32:01,056 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: status_report 2024-08-31 23:32:01,239 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 7 2024-08-31 23:32:01,239 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:01,240 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 7 2024-08-31 23:32:01,240 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:01,240 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 7 2024-08-31 23:32:01,969 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/config.yaml 2024-08-31 23:32:01,969 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:32:01,970 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:32:02,002 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: poll_exit 2024-08-31 23:32:03,208 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 8 2024-08-31 23:32:03,209 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: poll_exit 2024-08-31 23:32:03,209 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:03,210 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 8 2024-08-31 23:32:03,210 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:03,211 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 8 2024-08-31 23:32:03,211 INFO SenderThread:29126 [job_builder.py:build():443] Attempting to build job artifact 2024-08-31 23:32:03,212 INFO SenderThread:29126 [job_builder.py:_get_source_type():572] is repo sourced job 2024-08-31 23:32:03,212 WARNING SenderThread:29126 [job_builder.py:_log_if_verbose():274] No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job 2024-08-31 23:32:03,212 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 9 2024-08-31 23:32:03,213 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:03,213 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 9 2024-08-31 23:32:03,213 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:03,213 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 9 2024-08-31 23:32:03,213 INFO SenderThread:29126 [dir_watcher.py:finish():358] shutting down directory watcher 2024-08-31 23:32:03,970 INFO Thread-12 :29126 [dir_watcher.py:_on_file_modified():288] file/dir modified: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:32:03,971 INFO SenderThread:29126 [dir_watcher.py:finish():388] scan: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files 2024-08-31 23:32:03,971 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/config.yaml config.yaml 2024-08-31 23:32:03,972 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/diff.patch diff.patch 2024-08-31 23:32:03,972 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-metadata.json wandb-metadata.json 2024-08-31 23:32:03,972 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/requirements.txt requirements.txt 2024-08-31 23:32:03,972 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log output.log 2024-08-31 23:32:03,972 INFO SenderThread:29126 [dir_watcher.py:finish():402] scan save: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json wandb-summary.json 2024-08-31 23:32:03,972 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 10 2024-08-31 23:32:03,984 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:03,984 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 10 2024-08-31 23:32:03,985 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:03,985 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 10 2024-08-31 23:32:03,985 INFO SenderThread:29126 [file_pusher.py:finish():169] shutting down file pusher 2024-08-31 23:32:04,004 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: poll_exit 2024-08-31 23:32:04,004 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: poll_exit 2024-08-31 23:32:04,389 INFO wandb-upload_0:29126 [upload_job.py:push():130] Uploaded file /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/wandb-summary.json 2024-08-31 23:32:04,396 INFO wandb-upload_3:29126 [upload_job.py:push():130] Uploaded file /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/requirements.txt 2024-08-31 23:32:04,726 INFO wandb-upload_1:29126 [upload_job.py:push():130] Uploaded file /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/output.log 2024-08-31 23:32:04,751 INFO wandb-upload_2:29126 [upload_job.py:push():130] Uploaded file /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/files/config.yaml 2024-08-31 23:32:04,952 INFO Thread-11 (_thread_body):29126 [sender.py:transition_state():626] send defer: 11 2024-08-31 23:32:04,953 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:04,953 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 11 2024-08-31 23:32:04,954 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:04,954 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 11 2024-08-31 23:32:04,954 INFO SenderThread:29126 [file_pusher.py:join():175] waiting for file pusher 2024-08-31 23:32:04,955 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 12 2024-08-31 23:32:04,955 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:04,955 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 12 2024-08-31 23:32:04,956 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:04,956 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 12 2024-08-31 23:32:04,956 INFO SenderThread:29126 [file_stream.py:finish():601] file stream finish called 2024-08-31 23:32:05,005 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: poll_exit 2024-08-31 23:32:05,407 INFO SenderThread:29126 [file_stream.py:finish():605] file stream finish is done 2024-08-31 23:32:05,407 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 13 2024-08-31 23:32:05,407 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: poll_exit 2024-08-31 23:32:05,408 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:05,408 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 13 2024-08-31 23:32:05,409 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:05,409 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 13 2024-08-31 23:32:05,409 INFO SenderThread:29126 [sender.py:transition_state():626] send defer: 14 2024-08-31 23:32:05,409 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: defer 2024-08-31 23:32:05,410 INFO HandlerThread:29126 [handler.py:handle_request_defer():184] handle defer: 14 2024-08-31 23:32:05,410 DEBUG SenderThread:29126 [sender.py:send():391] send: final 2024-08-31 23:32:05,411 DEBUG SenderThread:29126 [sender.py:send():391] send: footer 2024-08-31 23:32:05,411 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: defer 2024-08-31 23:32:05,411 INFO SenderThread:29126 [sender.py:send_request_defer():622] handle sender defer: 14 2024-08-31 23:32:05,412 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: poll_exit 2024-08-31 23:32:05,413 DEBUG SenderThread:29126 [sender.py:send_request():418] send_request: poll_exit 2024-08-31 23:32:05,414 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: internal_messages 2024-08-31 23:32:05,416 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: get_summary 2024-08-31 23:32:05,417 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: sampled_history 2024-08-31 23:32:05,423 DEBUG HandlerThread:29126 [handler.py:handle_request():158] handle_request: shutdown 2024-08-31 23:32:05,423 INFO HandlerThread:29126 [handler.py:finish():884] shutting down handler 2024-08-31 23:32:06,413 INFO WriterThread:29126 [datastore.py:close():296] close: /workspace/nanoT5/outputs/2024-08-30/19-59-22/wandb/run-20240830_195924-mao0tqjy/run-mao0tqjy.wandb 2024-08-31 23:32:06,414 INFO SenderThread:29126 [sender.py:finish():1631] shutting down sender 2024-08-31 23:32:06,414 INFO SenderThread:29126 [file_pusher.py:finish():169] shutting down file pusher 2024-08-31 23:32:06,414 INFO SenderThread:29126 [file_pusher.py:join():175] waiting for file pusher