Llama.cpp Now Supports Qwen2-VL (Vision Language Model)

…

File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.py”, line 406, in hf_raise_for_status
response.raise_for_status()
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/requests/models.py”, line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/Qwen2-VL-2B-Instruct/resolve/main/processor_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py”, line 403, in cached_file
resolved_file=hf_hub_download(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py”, line 114, in _inner_fn
return fn(*args, **kwargs)
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 862, in hf_hub_download
return _hf_hub_download_to_cache_dir(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 969, in _hf_hub_download_to_cache_dir
_raise_on_head_call_error(head_call_error, force_download, local_files_only)
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 1484, in _raise_on_head_call_error
raise head_call_error
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 1376, in _get_metadata_or_catch_error
metadata=get_hf_file_metadata(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py”, line 114, in _inner_fn
return fn(*args, **kwargs)
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 1296, in get_hf_file_metadata
r=_request_wrapper(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 277, in _request_wrapper
response=_request_wrapper(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py”, line 301, in _request_wrapper
hf_raise_for_status(response)
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.py”, line 454, in hf_raise_for_status
raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-675de5d3-319c681e02ab26174d878b71;82815812-2512-468e-bba8-8beac819dd0c)

Repository Not Found for url: https://huggingface.co/Qwen2-VL-2B-Instruct/resolve/main/processor_config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/whatever/llama.cpp/examples/llava/qwen2_vl_surgery.py”, line 158, in
main(args)
File “/whatever/llama.cpp/examples/llava/qwen2_vl_surgery.py”, line 142, in main
processor: Qwen2VLProcessor=AutoProcessor.from_pretrained(model_name)
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/models/auto/processing_auto.py”, line 254, in from_pretrained
processor_config_file=get_file_from_repo(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py”, line 557, in get_file_from_repo
return cached_file(
File “/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py”, line 426, in cached_file
raise EnvironmentError(
OSError: Qwen2-VL-2B-Instruct is not a local folder and is not a valid model identifier listed on ‘https://huggingface.co/models’
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=`”>

(venv) ali0une@Debian:~/compil/llama.cpp$ python examples/llava/qwen2_vl_surgery.py "/whatever/Qwen/Qwen2-VL-2B-Instruct"
model_name:  /whatever/Qwen/Qwen2-VL-2B-Instruct
`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the `config` argument. All other arguments will be removed in v4.46
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00,  2.17it/s]
Qwen2VLVisionConfig {
  "depth": 32,
  "embed_dim": 1280,
  "hidden_act": "quick_gelu",
  "hidden_size": 1536,
  "in_channels": 3,
  "in_chans": 3,
  "mlp_ratio": 4,
  "model_type": "qwen2_vl",
  "num_heads": 16,
  "patch_size": 14,
  "spatial_merge_size": 2,
  "spatial_patch_size": 14,
  "temporal_patch_size": 2,
  "transformers_version": "4.47.0"
}

[to_gguf_name] vision_model.blocks.0.norm1.weight --> v.blk.0.ln1.weight

...

[to_gguf_name] merger.mlp.2.bias --> mm.2.bias
Traceback (most recent call last):

  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
    response.raise_for_status()
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/Qwen2-VL-2B-Instruct/resolve/main/processor_config.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 403, in cached_file
    resolved_file=hf_hub_download(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 862, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 969, in _hf_hub_download_to_cache_dir
    _raise_on_head_call_error(head_call_error, force_download, local_files_only)
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1484, in _raise_on_head_call_error
    raise head_call_error
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1376, in _get_metadata_or_catch_error
    metadata=get_hf_file_metadata(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1296, in get_hf_file_metadata
    r=_request_wrapper(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 277, in _request_wrapper
    response=_request_wrapper(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 301, in _request_wrapper
    hf_raise_for_status(response)
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 454, in hf_raise_for_status
    raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-675de5d3-319c681e02ab26174d878b71;82815812-2512-468e-bba8-8beac819dd0c)

Repository Not Found for url: https://huggingface.co/Qwen2-VL-2B-Instruct/resolve/main/processor_config.json.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/whatever/llama.cpp/examples/llava/qwen2_vl_surgery.py", line 158, in 
    main(args)
  File "/whatever/llama.cpp/examples/llava/qwen2_vl_surgery.py", line 142, in main
    processor: Qwen2VLProcessor=AutoProcessor.from_pretrained(model_name)
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/models/auto/processing_auto.py", line 254, in from_pretrained
    processor_config_file=get_file_from_repo(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 557, in get_file_from_repo
    return cached_file(
  File "/whatever/llama.cpp/venv/lib/python3.10/site-packages/transformers/utils/hub.py", line 426, in cached_file
    raise EnvironmentError(
OSError: Qwen2-VL-2B-Instruct is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=`