Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

86 Open 259 Closed
86 Open 259 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Feature] MMLU-PRO
#1161 opened May 15, 2024 by Ezra-Yu
1 task
@liushz
the prompt for triviaqa dataset
#1158 opened May 15, 2024 by mengrusun
1 task done
@liushz
[Bug] run with transformers==4.40.2, error "HuggingFacewithChatTemplate does not support ppl-based evaluation".
#1157 opened May 15, 2024 by zhulinJulia24
2 tasks done
@Leymore
[Feature] AGIEval数据集是否可以支持few-shot评测设置？
#1150 opened May 14, 2024 by cabbagecabbage
1 task
@bittersweet1999
可以创建一个微信群吗？
#1149 opened May 14, 2024 by bank010
1 task
@bittersweet1999
1
[Bug] math_gen数据集评估随机失败
#1148 opened May 14, 2024 by berton820
2 tasks done
@bittersweet1999
4
[Feature] Typos in the official document
#1147 opened May 13, 2024 by Galaxy-Husky
1 task done
@Leymore
1
storycloze_gen 测试不通过，显示torch.distributed.elastic.multiprocessing.api.SignalException [Feature]
#1142 opened May 12, 2024 by dh12306
1 task
CMB + Qwen1.5-72B-Chat got empty answers
#1141 opened May 11, 2024 by qy1026
2 tasks done
@tonysy
1
cmb_gen [bug]
#1137 opened May 11, 2024 by songyao199681
2 tasks done
@tonysy
官网上每个数据集所使用的config文件的版本如何知道？
#1130 opened May 9, 2024 by jyshee
@kennymckormick
1
新老版本的结果差距
#1126 opened May 9, 2024 by mengrusun
2 tasks done
@kennymckormick
1
[Bug] Base classes use @abstractmethod without subclassing from abc.ABC
#1124 opened May 8, 2024 by acylam
2 tasks done
@Leymore @liushz
[Improvement] Mutiprocess Evaluation time Bug
#1115 opened May 6, 2024 by wdndev
2 tasks done
@Leymore
1
[Feature] 请问在使用VLLM测评模型humaneval时，batch_size 不同导致 测评结果有区别是为什么？
#1097 opened Apr 26, 2024 by noforit
1 task
@tonysy
[Feature] ollama发布的模型如何评估支持
#1095 opened Apr 26, 2024 by 18600709862
1 task
@tonysy
1
[Bug] 不支持flores200数据集评测
#1083 opened Apr 24, 2024 by Liukairong2023
2 tasks done
@liushz
4
[Feature] 是否可以对arc-e和arc-c的base模型评估增加25-shot with choice
#1082 opened Apr 24, 2024 by siftxxx
1 task
@liushz
[Bug] There is a large gap in the partial subset of the long text evaluation data set
#1061 opened Apr 19, 2024 by bullw
2 tasks done
@tonysy
[Bug] DLCRunner无法获取运行结果
#1058 opened Apr 18, 2024 by Ethan-9606
2 tasks done
@kennymckormick
[Feature]在model文件中没有找到关于Qwen1.5-32b-chat的配置文件
#1057 opened Apr 18, 2024 by Egber1t
1 task
@kennymckormick
2
[Feature] 兼容torch_npu
#1045 opened Apr 12, 2024 by li126com
1 task
@tonysy
1
[Feature] 自定义数据集命令如何修改评估指标？
#1044 opened Apr 12, 2024 by liushiton
1 task
@tonysy
使用baichuan-7b评估humaneval数据与榜单差距过大
#1038 opened Apr 11, 2024 by zh190920
2 tasks done
@kennymckormick
2
[Bug] Long text evaluation parameters are not clear
#1035 opened Apr 10, 2024 by bullw
2 tasks done
@liushz
3
ProTip! Type g i on any issue or pull request to go back to the issue listing page.