3 .. currentmodule:: moe.eval
11 :func:`Eval.init` is run before the main pipeline and does the following:
16 \ Set builtins `HOME`, `CONFIG`, `LOG`, `DEBUG_LEVEL`, `VERBOSE`, also (temp?) `TDIR`, `USER_LOG`
17 \ Apply command-line overrides
18 \ Fix `CONFIG` and load the config file `CONFIG`
19 \ Fix `LOG`, `USER_LOG`, `VERBOSE`, `HOME`, `DEBUG_LEVEL`, `TDIR`
20 \ Start logging system (to `LOG` and `USER_LOG`)
21 \ Insert predefined hooks ([m]) and custom `HOOKS` to main pipeline
27 Ran by :func:`Eval.run`.
32 5 [m :func:`Eval.hook_init_dirs`] Check basic settings and directories
33 10 *Task must be ready (rsynced, pulled)*
34 15 [m :func:`Eval.hook_load_task_config`] Load task config file
35 20 [m :func:`Eval.hook_init_tasktype`] Import and initialize the task-type module
38 25 [tt] Locate the source
39 30 [tt] Initialize sandbox module
40 35 [tt] Initialize compiler module, insert its hooks
41 50 [cpl] Compile the source, perhaps using the sandbox
42 60 [tt] Run test pipeline for each `TEST` of `TESTS`
45 25 [tt] Locate the data files (as a dir)
46 30 [tt] Initialize sandbox module (if needed)
47 40 [tt] Initialize judge module, insert its hooks
48 60 [tt] Run test pipeline for each `TEST` of `TESTS`
51 70 Any failure jumps to this time
52 80 Cleanup *must* be finished
53 90 [m :func:`Eval.hook_write_metadata`] Write metafile
59 ======= ========================
61 ======= ========================
62 \ Tee logging to test log file
64 --------------------------------
65 10 [tt] Copy test data to `TDIR` and to `BOXDIR`
66 30 [tt] Run testee (inside sandbox)
67 40 [tt] Run judge (inside sandbox)
68 50 [tt] Read/parse judge result
69 60 [tt] Copy program output to `TDIR`
71 --------------------------------
72 10 [tt] Copy test data to `TDIR` and to `BOXDIR`
73 20 [tt] Locate the data file, copy to
74 40 [tt] Run judge (inside sandbox)
75 50 [tt] Read/parse judge result
77 --------------------------------
78 70 Any failure jumps to this time
79 80 [*] Cleanup must be finished
81 ======= ========================