Releases · open-compass/VLMEvalKit

What's Changed

[Feature] Support multi_generate by @kennymckormick in #1
[Tool] Minor updates 1205 by @kennymckormick in #3
TranCore-M 20231208 by @PCIResearch in #8
[Result] update TransCore Results by @kennymckormick in #9
COREMM Evaluation Benchmark by @youngfly11 in #6
Adding MMVet by @llllIlllll in #7
[Doc] README Update 12.11 by @kennymckormick in #10
Updating mmvet_eval by @llllIlllll in #11
[Feature] Add run.py and simplify the evaluation. by @kennymckormick in #12
[Doc] Refine README by @kennymckormick in #13
[Fix] Fix README by @kennymckormick in #14
add dataset md5 check integrity by @FangXinyu-0913 in #15
[Feature] Support two Vision APIs by @kennymckormick in #19
Add COCO dataset by @FangXinyu-0913 in #16
[Fix] Fix 1221 by @kennymckormick in #21
[Feature] More Robust API Evaluation by @kennymckormick in #22
[Refactor] Refactor Custom Prompt & Fix mPLUG-Owl2 acc by @kennymckormick in #23
[Dataset] VQA Datasets by @kennymckormick in #25
[Fix] Bug Fix by @kennymckormick in #26
Add MMMU dataset by @llllIlllll in #18
Add QwenVLPlus API by @llllIlllll in #27
[Result] Update MMMU Acc by @kennymckormick in #30
[Feature] Support LLaVA_XTuner models by @LZHgrla in #17
[Result] Update XTuner Performance by @kennymckormick in #31
[Result] Update COCO Caption Results by @kennymckormick in #35
add dataset ChartQA by @FangXinyu-0913 in #28
[Feature]: Add ScienceQA by @YuanLiuuuuuu in #24
[Dataset] MathVista dataset by @llllIlllll in #29
[Dataset] HallusionBench by @kennymckormick in #38
Add sharedcaptioner and cogvlm by @fitzpchao in #37
[Fix] Fix GPT error with parallel calling by @kennymckormick in #40
[Eval] multiple_choice.py: E->Z by @kennymckormick in #41
[Eval] Use exact matching for Y/N and multi-choice when OPENAI_API_KEY not set by @kennymckormick in #44
add VLM: monkey by @ShuoZhang2003 in #45
[Fix] Fix multiple choice evaluation when OPENAI_API_KEY missing by @kennymckormick in #48
[Improvement] Support non-contiguous choices by @kennymckormick in #49
Add Emu2 and Emu2_chat by @llllIlllll in #47
[Dataset] Add DocVQA by @llllIlllll in #50
[Benchmark] support AI2D by @kennymckormick in #51
add monkey-chat by @ShuoZhang2003 in #54
Add LLaVA-InternLM2 by @LZHgrla in #53
[Dataset] Support LLaVABench by @kennymckormick in #55
Support torchrun for emu2&emu2_chat and fix bug by @llllIlllll in #52
support sharegpt4v-13b by @xiaoachen98 in #56
Fix bug in file.py by @Ezra-Yu in #58
[Result] Update Evaluation Results by @kennymckormick in #60