Refactor MLLM #529
Refactor MLLM #529
Conversation
| val_dataset = [ | ||
| dict( | ||
| type=MMELLaVADataset, | ||
| data_file='/mnt/petrelfs/huanghaian/code/xtuner/LMUData/MME.tsv', |
There was a problem hiding this comment.
像这种路径，是让用户修改config来设置，还是说像VLMEvalKit一样，内部自动下载
There was a problem hiding this comment.
我觉得不用这么智能吧，直接要他们提前下载好，省的出现一些奇怪问题
| # set log processor | ||
| log_processor = dict(by_epoch=False) | ||
|
|
||
| # ==================== val and test cfg ======================= |
There was a problem hiding this comment.
此部分，是放到上面 PART 3 Dataset & Dataloader 还是单独放在下面？
There was a problem hiding this comment.
我觉得单独放下面，用户用起来更方便？
| print_log('============================================', 'current') | ||
| print_log(score, 'current') | ||
| print_log('============================================', 'current') | ||
| print_log(f'YOrN_eval successfully finished evaluating', 'current') |
There was a problem hiding this comment.
| print_log(f'YOrN_eval successfully finished evaluating', 'current') | |
| print_log('Hallusion successfully finished evaluating', 'current') |
| with pd.ExcelWriter(osp.join(work_dir, self.results_xlsx_path), engine='openpyxl') as writer: | ||
| results_df.to_excel(writer, index=False) | ||
|
|
||
| score = Hallusion_rating(data) |
There was a problem hiding this comment.
把 Hallusion_rating 函数放到本文件？ 因为这个应该只有这里会用到，所以方便查看。
| with pd.ExcelWriter(osp.join(work_dir, self.results_xlsx_path), engine='openpyxl') as writer: | ||
| results_df.to_excel(writer, index=False) | ||
|
|
||
| score = MME_rating(data) |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| template = prompt_template | ||
| self.template = template | ||
|
|
There was a problem hiding this comment.
| template = prompt_template | |
| self.template = template | |
| self.template = prompt_template | |
| results = collect_results(results, len(dataset)) | ||
| self.runner.logger.info('========= Starting the evaluation of a data ===========') | ||
| if is_main_process(): | ||
| metric = dataset.evaluate(results, self.runner.work_dir) |
There was a problem hiding this comment.
self.runner.work_dir 是否考虑携带 iter 信息，类似这里
xtuner/xtuner/engine/hooks/evaluate_chat_hook.py
Lines 93 to 98 in 0b5708c
| print_log('Samples: {}, Accuracy: {:.2f}%'.format(len(pred_list), acc), 'current') | ||
| print_log('============================================', 'current') | ||
| print_log(f'TextVQA successfully finished evaluating', 'current') | ||
| return {'acc': acc} |
There was a problem hiding this comment.
规定必须要返回字典吧，我记得，因为到时候 bset checkpoint 时候会读取
| @@ -96,7 +97,11 @@ def main(): | |||
| runner = RUNNERS.build(cfg) | |||
|
|
|||
| state_dict = guess_load_checkpoint(args.checkpoint) | |||
There was a problem hiding this comment.
checkpoint 现在是一个 optional argument，看起来是之前残留的bug，最好改成 positional argument。
In order to enhance the flexibility and usability of the multimodal model, it is necessary to refactor the existing llava code.