| 2022-01-14 22:15:34,709 INFO MainThread:7834 [internal.py:wandb_internal():87] W&B internal server running at pid: 7834, started at: 2022-01-14 22:15:34.709583 | |
| 2022-01-14 22:15:34,711 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: check_version | |
| 2022-01-14 22:15:34,712 INFO WriterThread:7834 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb | |
| 2022-01-14 22:15:34,712 DEBUG SenderThread:7834 [sender.py:send():234] send: header | |
| 2022-01-14 22:15:34,712 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: check_version | |
| 2022-01-14 22:15:34,785 DEBUG SenderThread:7834 [sender.py:send():234] send: run | |
| 2022-01-14 22:15:34,980 INFO SenderThread:7834 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files | |
| 2022-01-14 22:15:34,980 INFO SenderThread:7834 [sender.py:_start_run_threads():804] run started: 24dma583 with start time 1642198533 | |
| 2022-01-14 22:15:34,980 DEBUG SenderThread:7834 [sender.py:send():234] send: summary | |
| 2022-01-14 22:15:34,980 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-14 22:15:34,981 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: run_start | |
| 2022-01-14 22:15:35,985 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json | |
| 2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:__init__():40] meta init | |
| 2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:__init__():54] meta init done | |
| 2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:probe():214] probe | |
| 2022-01-14 22:15:37,286 DEBUG HandlerThread:7834 [meta.py:_setup_git():204] setup git | |
| 2022-01-14 22:15:37,315 DEBUG HandlerThread:7834 [meta.py:_setup_git():211] setup git done | |
| 2022-01-14 22:15:37,315 DEBUG HandlerThread:7834 [meta.py:_save_code():92] save code | |
| 2022-01-14 22:15:37,326 DEBUG HandlerThread:7834 [meta.py:_save_code():113] save code done | |
| 2022-01-14 22:15:37,326 DEBUG HandlerThread:7834 [meta.py:_save_patches():130] save patches | |
| 2022-01-14 22:15:37,985 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py | |
| 2022-01-14 22:15:37,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code | |
| 2022-01-14 22:15:39,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:15:40,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch | |
| 2022-01-14 22:15:42,987 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:15:45,607 ERROR HandlerThread:7834 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds | |
| 2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_patches():172] save patches done | |
| 2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_pip():58] save pip | |
| 2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_pip():72] save pip done | |
| 2022-01-14 22:15:45,608 DEBUG HandlerThread:7834 [meta.py:probe():252] probe done | |
| 2022-01-14 22:15:45,643 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:15:45,643 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:15:45,786 DEBUG SenderThread:7834 [sender.py:send():234] send: config | |
| 2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: config | |
| 2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: config | |
| 2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: files | |
| 2022-01-14 22:15:45,787 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now | |
| 2022-01-14 22:15:45,789 INFO SenderThread:7834 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now | |
| 2022-01-14 22:15:45,988 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json | |
| 2022-01-14 22:15:45,989 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt | |
| 2022-01-14 22:15:46,312 INFO Thread-12 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/2juok80v-code/run_mlm_flax.py | |
| 2022-01-14 22:15:46,330 INFO Thread-11 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/xnc44171-wandb-metadata.json | |
| 2022-01-14 22:15:50,990 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:15:59,991 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:15:59,991 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:16:05,368 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:16:05,997 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml | |
| 2022-01-14 22:16:15,132 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:16:15,132 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:16:30,272 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:16:30,272 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:16:35,439 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:16:39,009 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:16:45,408 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:16:45,408 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:17:00,601 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:17:00,601 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:17:05,512 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:17:15,756 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:17:15,756 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:17:30,970 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:17:30,971 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:17:35,586 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:17:46,135 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:17:46,136 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:18:01,309 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:18:01,309 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:18:05,663 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:18:16,458 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:18:16,458 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:18:31,596 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:18:31,597 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:18:35,742 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:18:46,731 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-14 22:18:46,732 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-14 22:19:03,067 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:19:03,953 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:03,953 DEBUG SenderThread:7834 [sender.py:send():234] send: telemetry | |
| 2022-01-14 22:19:03,953 DEBUG SenderThread:7834 [sender.py:send():234] send: exit | |
| 2022-01-14 22:19:03,953 INFO SenderThread:7834 [sender.py:send_exit():366] handling exit code: 1 | |
| 2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:send_exit():368] handling runtime: 208 | |
| 2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:send_exit():374] send defer | |
| 2022-01-14 22:19:03,954 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:03,955 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:03,955 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 0 | |
| 2022-01-14 22:19:03,955 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:03,955 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 0 | |
| 2022-01-14 22:19:03,955 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 1 | |
| 2022-01-14 22:19:03,955 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:03,955 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 1 | |
| 2022-01-14 22:19:04,011 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:04,011 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 1 | |
| 2022-01-14 22:19:04,011 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 2 | |
| 2022-01-14 22:19:04,011 DEBUG SenderThread:7834 [sender.py:send():234] send: stats | |
| 2022-01-14 22:19:04,012 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:04,012 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 2 | |
| 2022-01-14 22:19:04,012 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:04,012 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 2 | |
| 2022-01-14 22:19:04,012 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 3 | |
| 2022-01-14 22:19:04,012 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:04,012 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 3 | |
| 2022-01-14 22:19:04,012 DEBUG SenderThread:7834 [sender.py:send():234] send: summary | |
| 2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-14 22:19:04,013 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 3 | |
| 2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 4 | |
| 2022-01-14 22:19:04,013 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:04,013 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 4 | |
| 2022-01-14 22:19:04,013 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 4 | |
| 2022-01-14 22:19:04,057 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:04,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:19:04,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json | |
| 2022-01-14 22:19:04,198 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 5 | |
| 2022-01-14 22:19:04,198 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:04,198 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:04,198 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 5 | |
| 2022-01-14 22:19:04,199 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:04,199 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 5 | |
| 2022-01-14 22:19:04,199 INFO SenderThread:7834 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-01-14 22:19:04,300 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml | |
| 2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files | |
| 2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml config.yaml | |
| 2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch diff.patch | |
| 2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt requirements.txt | |
| 2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log output.log | |
| 2022-01-14 22:19:05,070 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json wandb-summary.json | |
| 2022-01-14 22:19:05,070 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json wandb-metadata.json | |
| 2022-01-14 22:19:05,072 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py code/run_mlm_flax.py | |
| 2022-01-14 22:19:05,073 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 6 | |
| 2022-01-14 22:19:05,073 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,081 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:05,081 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 6 | |
| 2022-01-14 22:19:05,081 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:05,081 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 6 | |
| 2022-01-14 22:19:05,081 INFO SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-14 22:19:05,183 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,183 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,285 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,285 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,387 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,387 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,488 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,489 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,539 INFO Thread-13 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml | |
| 2022-01-14 22:19:05,556 INFO Thread-15 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log | |
| 2022-01-14 22:19:05,561 INFO Thread-14 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt | |
| 2022-01-14 22:19:05,590 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,590 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,599 INFO Thread-16 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json | |
| 2022-01-14 22:19:05,692 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,692 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,794 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:05,794 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:05,799 INFO Thread-7 :7834 [sender.py:transition_state():387] send defer: 7 | |
| 2022-01-14 22:19:05,800 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:05,800 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 7 | |
| 2022-01-14 22:19:05,800 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:05,800 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 7 | |
| 2022-01-14 22:19:05,896 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:06,218 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 8 | |
| 2022-01-14 22:19:06,218 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:06,218 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:06,218 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 8 | |
| 2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:06,219 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 8 | |
| 2022-01-14 22:19:06,219 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 9 | |
| 2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send():234] send: final | |
| 2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send():234] send: footer | |
| 2022-01-14 22:19:06,219 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-14 22:19:06,220 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 9 | |
| 2022-01-14 22:19:06,220 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer | |
| 2022-01-14 22:19:06,220 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 9 | |
| 2022-01-14 22:19:06,320 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-14 22:19:06,320 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-14 22:19:06,320 INFO SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-14 22:19:06,598 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: get_summary | |
| 2022-01-14 22:19:06,618 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: sampled_history | |
| 2022-01-14 22:19:06,619 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: shutdown | |
| 2022-01-14 22:19:06,619 INFO HandlerThread:7834 [handler.py:finish():731] shutting down handler | |
| 2022-01-14 22:19:07,220 INFO WriterThread:7834 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb | |
| 2022-01-14 22:19:07,575 INFO SenderThread:7834 [sender.py:finish():1070] shutting down sender | |
| 2022-01-14 22:19:07,575 INFO SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-14 22:19:07,576 INFO SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-14 22:19:07,578 INFO MainThread:7834 [internal.py:handle_exit():77] Internal process exited | |