[twitter] x.com links give Unsupported URL error #9923

InklingGirl · 2024-05-15T10:47:33Z

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

I'm reporting that yt-dlp is broken on a supported site
I've verified that I have updated yt-dlp to nightly or master (update instructions)
I've checked that all provided URLs are playable in a browser with the same IP and same login details
I've checked that all URLs and arguments with special characters are properly quoted or escaped
I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
I've read the guidelines for opening an issue
I've read about sharing account credentials and I'm willing to share it if required

Region

Australia

Provide a description that is worded well enough to be understood

https://twitter.com has been changed to https://x.com, but yt-dlp is still performing a redirect to the former URL it does not need to, which seems to break the download.

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
If using API, add 'verbose': True to YoutubeDL params instead
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://x.com/historyinmemes/status/1790637656616943991']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.04.09 from yt-dlp/yt-dlp [ff0779267] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.04.09 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.04.09 from yt-dlp/yt-dlp)
[generic] Extracting URL: https://x.com/historyinmemes/status/1790637656616943991
[generic] 1790637656616943991: Downloading webpage
[redirect] Following redirect to https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
[generic] Extracting URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
[generic] migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
Traceback (most recent call last):
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1606, in wrapper
    return func(self, *args, **kwargs)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1741, in __extract_info
    ie_result = ie.extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 2514, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P

The text was updated successfully, but these errors were encountered:

joe-oli · 2024-05-15T13:15:04Z

I confirm yt-dlp no longer works on X/Twitter; just started happening very recently (approximately 14/May/2024), which is most likely related to the url for X is now x.com instead of twitter.com

bashonly · 2024-05-15T13:33:16Z

Internally, the site still uses twitter domains. All we need to do is add x.com to the _VALID_URLs

Closes #9923 Authored by: bashonly

InklingGirl · 2024-05-17T03:05:49Z

Will this update be pushed to pip soon?

bashonly · 2024-05-17T03:06:39Z

@InklingGirl already has: python -m pip install -U --pre "yt-dlp[default]"

jshir · 2024-05-18T01:39:31Z

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _________________: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

ggous · 2024-05-20T06:01:21Z

I have updated : sudo yt-dlp -U

and I am receiving:

ERROR: Unsupported URL: https://x.com/i/status/1792192732015800675

bashonly · 2024-05-20T06:24:36Z

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

ggous · 2024-05-20T06:42:42Z

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

Thanks! It works now!

hsn10 · 2024-05-20T11:03:15Z

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

Grub4K · 2024-05-20T11:08:21Z

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

Use yt-dlp --update-to nightly, download the nightly build manually or use pip install --pre "yt-dlp[default]".

InklingGirl · 2024-05-25T19:49:25Z

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _________________: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

same here @jshir

bashonly · 2024-05-25T19:55:39Z

show verbose log please @InklingGirl

InklingGirl · 2024-05-25T20:07:35Z

show verbose log please @InklingGirl

@bashonly this happens w/ any video that previously required & worked w/ cookies:

[debug] Command-line config: ['--cookies', 'Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.16.232713 from yt-dlp/yt-dlp-nightly-builds [6d8a53d87] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1803 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/latest/download/_update_spec
Current version: nightly@2024.05.16.232713 from yt-dlp/yt-dlp-nightly-builds
Latest version: nightly@2024.05.23.232707 from yt-dlp/yt-dlp-nightly-builds
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1428, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1406, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1283, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1233, in raise_login_required
    raise ExtractorError(msg, expected=True)

bashonly · 2024-05-25T20:18:33Z

@InklingGirl update. Your (old) nightly version does not have 3e35aa3, which fixed that issue

python3 -m pip install --pre -U "yt-dlp[default]"

InklingGirl · 2024-05-25T21:51:56Z

I upgraded to it & get the exact same result.

EDIT: I generated a new cookies file & that solved the issue. The old cookies must have been for twitter.com instead of x.com & that was what caused the problem.

InklingGirl · 2024-05-28T10:23:56Z

& it seems to be broken again.

bashonly · 2024-05-28T14:06:58Z

@InklingGirl please share a verbose log

InklingGirl · 2024-05-28T14:26:35Z

[debug] Command-line config: ['--cookies', '/home/christian/Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)

bashonly · 2024-05-28T14:29:32Z

What happens when you try with --cookies-from-browser?

InklingGirl · 2024-05-28T15:04:04Z

Is that supposed to work w/ an apt-get installation of brave-browser on Ubuntu 22?

bashonly · 2024-05-28T15:05:27Z

it should yeah

InklingGirl · 2024-05-28T15:10:25Z

yt-dlp: error: unsupported browser specified for cookies: "https". Supported browsers are: brave, chrome, chromium, edge, firefox, opera, safari, vivaldi, whale

InklingGirl · 2024-05-28T15:11:37Z

I'm assuming I had to specify the browser, here's the verbose log for that:

[debug] Command-line config: ['--cookies-from-browser=brave', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
Extracting cookies from brave
[debug] Extracting cookies from: "/home/christian/.config/BraveSoftware/Brave-Browser/Default/Cookies"
[Cookies] Loading cookie      0/  2019XDG_CURRENT_DESKTOP is set to an unknown value: "ubuntu"
[debug] detected desktop environment: OTHER
[debug] Chosen keyring: BASICTEXT
WARNING: cannot decrypt v11 cookies: no key found
Extracted 44 cookies from brave (1975 could not be decrypted)
[debug] cookie version breakdown: {'v10': 0, 'v11': 1975, 'other': 0, 'unencrypted': 44}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)

InklingGirl added site-bug Issue with a specific website triage Untriaged issue labels May 15, 2024

bashonly removed the triage Untriaged issue label May 15, 2024

bashonly mentioned this issue May 15, 2024

[ie/twitter] Support x.com URLs #9926

Merged

5 tasks

bashonly closed this as completed in #9926 May 16, 2024

bashonly added a commit that referenced this issue May 16, 2024

[ie/twitter] Support x.com URLs (#9926)

4813173

Closes #9923 Authored by: bashonly

bashonly changed the title ~~URL Change to X (Formely Twitter) Seems to Break Support~~ [twitter] x.com links give Unsupported URL error May 17, 2024

This was referenced May 17, 2024

🚨[IMPORTANT]🚨 KNOWN ISSUES/FAQ #3766

Open

X/Twitter video URLs no longer work #9950

Closed

This was referenced May 18, 2024

[ie/twitter] Fix auth for x.com migration #9952

Merged

x.com (works with Twitter.com already) #9972

Closed

This was referenced May 22, 2024

not support x.com video #9998

Closed

twitter changed domain name to x.com #10010

Closed

x.com (twitter) gives "unsupported URL" #10014

Closed

StefanLobbenmeier mentioned this issue May 25, 2024

Downloads don't work on X videos StefanLobbenmeier/youtube-dl-gui#123

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[twitter] x.com links give Unsupported URL error #9923

[twitter] x.com links give Unsupported URL error #9923

InklingGirl commented May 15, 2024

joe-oli commented May 15, 2024

bashonly commented May 15, 2024

InklingGirl commented May 17, 2024

bashonly commented May 17, 2024

jshir commented May 18, 2024

ggous commented May 20, 2024

bashonly commented May 20, 2024 •

edited

ggous commented May 20, 2024

hsn10 commented May 20, 2024

Grub4K commented May 20, 2024

InklingGirl commented May 25, 2024

bashonly commented May 25, 2024

InklingGirl commented May 25, 2024 •

edited

bashonly commented May 25, 2024 •

edited

InklingGirl commented May 25, 2024 •

edited

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

InklingGirl commented May 28, 2024

[twitter] x.com links give Unsupported URL error #9923

[twitter] x.com links give Unsupported URL error #9923

Comments

InklingGirl commented May 15, 2024

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Region

Provide a description that is worded well enough to be understood

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

joe-oli commented May 15, 2024

bashonly commented May 15, 2024

InklingGirl commented May 17, 2024

bashonly commented May 17, 2024

jshir commented May 18, 2024

ggous commented May 20, 2024

bashonly commented May 20, 2024 • edited

ggous commented May 20, 2024

hsn10 commented May 20, 2024

Grub4K commented May 20, 2024

InklingGirl commented May 25, 2024

bashonly commented May 25, 2024

InklingGirl commented May 25, 2024 • edited

bashonly commented May 25, 2024 • edited

InklingGirl commented May 25, 2024 • edited

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

bashonly commented May 28, 2024

InklingGirl commented May 28, 2024

InklingGirl commented May 28, 2024

bashonly commented May 20, 2024 •

edited

InklingGirl commented May 25, 2024 •

edited

bashonly commented May 25, 2024 •

edited

InklingGirl commented May 25, 2024 •

edited