File size: 21,861 Bytes
9c3de9e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 |
2022-01-14 22:15:34,709 INFO MainThread:7834 [internal.py:wandb_internal():87] W&B internal server running at pid: 7834, started at: 2022-01-14 22:15:34.709583
2022-01-14 22:15:34,711 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: check_version
2022-01-14 22:15:34,712 INFO WriterThread:7834 [datastore.py:open_for_write():77] open: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb
2022-01-14 22:15:34,712 DEBUG SenderThread:7834 [sender.py:send():234] send: header
2022-01-14 22:15:34,712 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: check_version
2022-01-14 22:15:34,785 DEBUG SenderThread:7834 [sender.py:send():234] send: run
2022-01-14 22:15:34,980 INFO SenderThread:7834 [dir_watcher.py:__init__():169] watching files in: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files
2022-01-14 22:15:34,980 INFO SenderThread:7834 [sender.py:_start_run_threads():804] run started: 24dma583 with start time 1642198533
2022-01-14 22:15:34,980 DEBUG SenderThread:7834 [sender.py:send():234] send: summary
2022-01-14 22:15:34,980 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:15:34,981 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: run_start
2022-01-14 22:15:35,985 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:__init__():40] meta init
2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:__init__():54] meta init done
2022-01-14 22:15:37,284 DEBUG HandlerThread:7834 [meta.py:probe():214] probe
2022-01-14 22:15:37,286 DEBUG HandlerThread:7834 [meta.py:_setup_git():204] setup git
2022-01-14 22:15:37,315 DEBUG HandlerThread:7834 [meta.py:_setup_git():211] setup git done
2022-01-14 22:15:37,315 DEBUG HandlerThread:7834 [meta.py:_save_code():92] save code
2022-01-14 22:15:37,326 DEBUG HandlerThread:7834 [meta.py:_save_code():113] save code done
2022-01-14 22:15:37,326 DEBUG HandlerThread:7834 [meta.py:_save_patches():130] save patches
2022-01-14 22:15:37,985 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py
2022-01-14 22:15:37,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code
2022-01-14 22:15:39,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:40,986 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch
2022-01-14 22:15:42,987 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:45,607 ERROR HandlerThread:7834 [meta.py:_save_patches():171] Error generating diff: Command '['git', 'diff', '--submodule=diff', 'HEAD']' timed out after 5 seconds
2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_patches():172] save patches done
2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_pip():58] save pip
2022-01-14 22:15:45,607 DEBUG HandlerThread:7834 [meta.py:_save_pip():72] save pip done
2022-01-14 22:15:45,608 DEBUG HandlerThread:7834 [meta.py:probe():252] probe done
2022-01-14 22:15:45,643 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:15:45,643 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:15:45,786 DEBUG SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: config
2022-01-14 22:15:45,787 DEBUG SenderThread:7834 [sender.py:send():234] send: files
2022-01-14 22:15:45,787 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now
2022-01-14 22:15:45,789 INFO SenderThread:7834 [sender.py:_save_file():939] saving file code/run_mlm_flax.py with policy now
2022-01-14 22:15:45,988 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json
2022-01-14 22:15:45,989 INFO Thread-8 :7834 [dir_watcher.py:_on_file_created():217] file/dir created: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt
2022-01-14 22:15:46,312 INFO Thread-12 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/2juok80v-code/run_mlm_flax.py
2022-01-14 22:15:46,330 INFO Thread-11 :7834 [upload_job.py:push():137] Uploaded file /tmp/tmpxqv1l1fswandb/xnc44171-wandb-metadata.json
2022-01-14 22:15:50,990 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:15:59,991 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:15:59,991 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:05,368 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:16:05,997 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:16:15,132 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:15,132 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:30,272 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:30,272 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:16:35,439 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:16:39,009 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:16:45,408 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:16:45,408 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:00,601 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:00,601 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:05,512 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:17:15,756 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:15,756 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:30,970 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:30,971 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:17:35,586 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:17:46,135 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:17:46,136 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:01,309 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:01,309 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:05,663 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:18:16,458 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:16,458 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:31,596 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:31,597 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:18:35,742 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:18:46,731 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: stop_status
2022-01-14 22:18:46,732 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: stop_status
2022-01-14 22:19:03,067 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:03,953 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:03,953 DEBUG SenderThread:7834 [sender.py:send():234] send: telemetry
2022-01-14 22:19:03,953 DEBUG SenderThread:7834 [sender.py:send():234] send: exit
2022-01-14 22:19:03,953 INFO SenderThread:7834 [sender.py:send_exit():366] handling exit code: 1
2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:send_exit():368] handling runtime: 208
2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:19:03,954 INFO SenderThread:7834 [sender.py:send_exit():374] send defer
2022-01-14 22:19:03,954 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:03,955 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:03,955 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 0
2022-01-14 22:19:03,955 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:03,955 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 0
2022-01-14 22:19:03,955 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 1
2022-01-14 22:19:03,955 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:03,955 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 1
2022-01-14 22:19:04,011 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,011 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 1
2022-01-14 22:19:04,011 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 2
2022-01-14 22:19:04,011 DEBUG SenderThread:7834 [sender.py:send():234] send: stats
2022-01-14 22:19:04,012 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,012 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 2
2022-01-14 22:19:04,012 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,012 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 2
2022-01-14 22:19:04,012 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 3
2022-01-14 22:19:04,012 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,012 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 3
2022-01-14 22:19:04,012 DEBUG SenderThread:7834 [sender.py:send():234] send: summary
2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:_save_file():939] saving file wandb-summary.json with policy end
2022-01-14 22:19:04,013 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 3
2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 4
2022-01-14 22:19:04,013 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,013 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 4
2022-01-14 22:19:04,013 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,013 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 4
2022-01-14 22:19:04,057 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:04,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:04,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:19:04,198 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 5
2022-01-14 22:19:04,198 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:04,198 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:04,198 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 5
2022-01-14 22:19:04,199 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:04,199 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 5
2022-01-14 22:19:04,199 INFO SenderThread:7834 [dir_watcher.py:finish():283] shutting down directory watcher
2022-01-14 22:19:04,300 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,068 INFO Thread-8 :7834 [dir_watcher.py:_on_file_modified():230] file/dir modified: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():313] scan: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files
2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml config.yaml
2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/diff.patch diff.patch
2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt requirements.txt
2022-01-14 22:19:05,069 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log output.log
2022-01-14 22:19:05,070 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json wandb-summary.json
2022-01-14 22:19:05,070 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-metadata.json wandb-metadata.json
2022-01-14 22:19:05,072 INFO SenderThread:7834 [dir_watcher.py:finish():327] scan save: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/code/run_mlm_flax.py code/run_mlm_flax.py
2022-01-14 22:19:05,073 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 6
2022-01-14 22:19:05,073 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,081 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:05,081 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 6
2022-01-14 22:19:05,081 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:05,081 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 6
2022-01-14 22:19:05,081 INFO SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 22:19:05,183 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,183 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,285 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,285 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,387 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,387 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,488 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,489 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,539 INFO Thread-13 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/config.yaml
2022-01-14 22:19:05,556 INFO Thread-15 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/output.log
2022-01-14 22:19:05,561 INFO Thread-14 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/requirements.txt
2022-01-14 22:19:05,590 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,590 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,599 INFO Thread-16 :7834 [upload_job.py:push():137] Uploaded file /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/files/wandb-summary.json
2022-01-14 22:19:05,692 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,692 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,794 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:05,794 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:05,799 INFO Thread-7 :7834 [sender.py:transition_state():387] send defer: 7
2022-01-14 22:19:05,800 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:05,800 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 7
2022-01-14 22:19:05,800 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:05,800 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 7
2022-01-14 22:19:05,896 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:06,218 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 8
2022-01-14 22:19:06,218 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:06,218 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:06,218 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 8
2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:06,219 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 8
2022-01-14 22:19:06,219 INFO SenderThread:7834 [sender.py:transition_state():387] send defer: 9
2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send():234] send: final
2022-01-14 22:19:06,219 DEBUG SenderThread:7834 [sender.py:send():234] send: footer
2022-01-14 22:19:06,219 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: defer
2022-01-14 22:19:06,220 INFO HandlerThread:7834 [handler.py:handle_request_defer():147] handle defer: 9
2022-01-14 22:19:06,220 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: defer
2022-01-14 22:19:06,220 INFO SenderThread:7834 [sender.py:send_request_defer():383] handle sender defer: 9
2022-01-14 22:19:06,320 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: poll_exit
2022-01-14 22:19:06,320 DEBUG SenderThread:7834 [sender.py:send_request():248] send_request: poll_exit
2022-01-14 22:19:06,320 INFO SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 22:19:06,598 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: get_summary
2022-01-14 22:19:06,618 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: sampled_history
2022-01-14 22:19:06,619 DEBUG HandlerThread:7834 [handler.py:handle_request():130] handle_request: shutdown
2022-01-14 22:19:06,619 INFO HandlerThread:7834 [handler.py:finish():731] shutting down handler
2022-01-14 22:19:07,220 INFO WriterThread:7834 [datastore.py:close():281] close: /data/roberta-base-ncc/wandb/run-20220114_221533-24dma583/run-24dma583.wandb
2022-01-14 22:19:07,575 INFO SenderThread:7834 [sender.py:finish():1070] shutting down sender
2022-01-14 22:19:07,575 INFO SenderThread:7834 [file_pusher.py:finish():177] shutting down file pusher
2022-01-14 22:19:07,576 INFO SenderThread:7834 [file_pusher.py:join():182] waiting for file pusher
2022-01-14 22:19:07,578 INFO MainThread:7834 [internal.py:handle_exit():77] Internal process exited
|