Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add HF siglip vision encoder #9185

Open
wants to merge 74 commits into
base: main
Choose a base branch
from
Open

Add HF siglip vision encoder #9185

wants to merge 74 commits into from
+106 −107

Conversation

HuiyingLi
Copy link
Contributor

What does this PR do ?

  • HF Siglip support
  • steerlm label in template
  • inference minor fixes
  • handle CLEAN RESPONSE

Collection: [Note which collection this PR will affect]

Changelog

  • Add specific line by line info of high level changes in this PR.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

yaoyu-33 and others added 30 commits March 15, 2024 18:27
@yaoyu-33
temp save
f2f267d 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_seq_pack
b532d1b
@yaoyu-33
temp save 2
b0ace6b 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
update code
0020fe3 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_seq_pack
76fb748
@yaoyu-33
enable seq packing
a8f2248 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
fix neva and clip
9fab5a5 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Enable parallel seq packing algo and few other fixes
d8474fb 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_seq_pack
c56ec9b 
# Conflicts:
#	examples/multimodal/vision_language_foundation/clip/megatron_clip_pretrain.py
@yaoyu-33
Pipeline parallel support
e8a9a6d 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Update data preprocess
c5ffa83 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_pp_support
e11e260
@yaoyu-33
fix few pp issues
2bc5d66 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'yuya/neva_seq_pack' into yuya/neva_pp_support
4843e54
@yaoyu-33
enable sequence packing w/ PP
78034ce 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Fix cu_seqlens in inputs
8561e60 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
add assert
2ac6b27 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_pp_support
d138b1e 
# Conflicts:
#	nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py
@yaoyu-33
Depend on PP to decide whether do padding
5e6994d 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
0544758 
for more information, see https://pre-commit.ci
@yaoyu-33
Add docstring
6af32af 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/neva_pp_support' into yuya/…
cd513b3 
…neva_pp_support
@yaoyu-33
Fix few evaluation issues
3655f7d 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_pp_support
f54e565
@yaoyu-33
Fix few PP evaluation issues
4bb0313 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Address comments
6efa4fa 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
9c44e30 
for more information, see https://pre-commit.ci
@yaoyu-33
add llama3 template
37953d4 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
address comments
56700e7 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Fix license
b5a4c27 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33 and others added 12 commits May 3, 2024 13:18
@yaoyu-33
Allow dist checkpoint to be non-strict
9f6d98e 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Fix
3c4b5a1 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Some fixes for PP + dist ckpt in Neva
3dde1eb 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
fix peft
bbf0832 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
few fixes for lora
9f5d615 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_llama3
cb3ae62 
# Conflicts:
#	nemo/collections/multimodal/data/neva/neva_dataset.py
#	nemo/collections/nlp/parts/nlp_overrides.py
@yaoyu-33
checkpoint updates
4588b6a 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Apply isort and black reformatting
725e353 
Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>
@yaoyu-33
Merge branch 'main' into yuya/neva_llama3
7c38044
@yaoyu-33
Merge remote-tracking branch 'origin/yuya/neva_llama3' into yuya/neva…
9bd4929 
…_llama3
@yaoyu-33
bug fix
34bff23 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@HuiyingLi
Add HF siglip vision encoder
79f042b 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
HuiyingLi and others added 4 commits May 13, 2024 16:10
@HuiyingLi
handle steerlm label in nv_dpo template
14d4327 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@yaoyu-33
Add neva dist checkpoint converter
bcf6385 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Apply isort and black reformatting
24785dd 
Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>
@HuiyingLi
fix CLEAN RESPONSE logic to not use last EOS
b605b6e 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@github-actions github-actions bot added the NLP label May 13, 2024
@HuiyingLi
strip extra_id_1 from clean response
56a3bb4 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@HuiyingLi
change inference time image processor
72e75cb 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
yaoyu-33
yaoyu-33 reviewed May 14, 2024
View reviewed changes
nemo/collections/nlp/modules/common/text_generation_utils.py Outdated
@@ -196,7 +196,8 @@ def megatron_neva_generate(model, prompt_dict_list, length_params, sampling_para
clean_response = clean_response[last_match_end_position:]
clean_response = clean_response.strip("<extra_id_1>")
elif conv_template == 'nv_dpo':
clean_response = clean_response.split("<extra_id_1>")[-2][10:] # [10:] for removing "Assistant\n"
clean_response = clean_response.split("<extra_id_1>Assistant\n")[1]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean_response.split("<extra_id_1>Assistant\n")[-1]?
imagine there are multi rounds

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! resolved

yaoyu-33 and others added 7 commits May 14, 2024 10:16
@yaoyu-33
resolve comments
08dbfa5 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@HuiyingLi
remove open_clip vision encoder for siglip
1cf8480 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@yaoyu-33
update neva dist ckpt apis
968a75c 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33
Apply isort and black reformatting
0b93023 
Signed-off-by: yaoyu-33 <yaoyu-33@users.noreply.github.com>
@yaoyu-33
fix return
5e5ad9e 
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@HuiyingLi
resolve CLEAN RESPONSE multiturn issue
495ffc8 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@HuiyingLi
Merge branch 'yuya/neva_llama3' into siglip_support
2753d5d 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
@yaoyu-33 yaoyu-33 marked this pull request as ready for review May 22, 2024 22:24
@HuiyingLi
Merge branch 'main' into siglip_support
62a287e 
Signed-off-by: HuiyingLi <willwin.lee@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yaoyu-33 yaoyu-33 yaoyu-33 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Multi Modal NLP
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@HuiyingLi @yaoyu-33