Skip to content

Add user agent to curl in QPA example #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add user agent to curl in QPA example #60

wants to merge 1 commit into from

Conversation

clemisch
Copy link

Otherwise, download is blocked by Cloudflare:

$ curl https://www.iucr.org/__data/iucr/powder/QARR/col/cpd-1h.prn
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Arial,"Noto Sans",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;padding-left:1.5rem;max-width:60rem}@media (width <= 720px){.main-content{margin-top:4rem}}.h2{line-height:2.25rem;font-size:1.5rem;font-weight:500}@media (width <= 720px){.h2{line-height:1.5rem;font-size:1.25rem}}#challenge-error-text{background-image:url("data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+");background-repeat:no-repeat;background-size:contain;padding-left:34px}@media (prefers-color-scheme: dark){body{background-color:#222;color:#d9d9d9}}</style><meta http-equiv="refresh" content="360"></head><body><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script>(function(){window._cf_chl_opt = {cvId: '3',cZone: 'www.iucr.org',cType: 'managed',cRay: '966bc1814dbc6fc6',cH: 'ArovDcV8FNtWQfDwv.DV01IjdWut5YCKJ7YeSUlv6co-1753783086-1.2.1.1-bYpHMUY2KaBi2wwML5q_sxSmk6Y.Z4NuMgZwOWtMz6LYOm7PhLfM2EYE56_d1reL',cUPMDTk:"/__data/iucr/powder/QARR/col/cpd-1h.prn?__cf_chl_tk=odEgREBEC0ei4w.rTzee9t0TAHD.Fn8t_urGp2AcwPI-1753783086-1.0.1.1-AEoJRZPcfW14LaVgLV5Vr1kBwpbtLXY.iAeIt7wJmbA",cFPWv: 'b',cITimeS: '1753783086',cTplC:0,cTplV:5,cTplB: 'cf',fa:"/__data/iucr/powder/QARR/col/cpd-1h.prn?__cf_chl_f_tk=odEgREBEC0ei4w.rTzee9t0TAHD.Fn8t_urGp2AcwPI-1753783086-1.0.1.1-AEoJRZPcfW14LaVgLV5Vr1kBwpbtLXY.iAeIt7wJmbA",md: 'Clw4QMAlufNgO9aygJMnLCWVGvltyBx0xozFqhKVj20-1753783086-1.2.1.1-8cxXGOmX0EHt.F3LJXVd7LYPgycfZFRyy3gHKgsqv0wXIn4se7qsgLzvXOr0t.yzl21Z1vAWq.236Y_lJrX_jHpt95tFzdt9f5uy7ppgLSLueRKCmR9ONe2bQKaSnWJyVFXrNI80.Cg0hBfpEmX1iiw_T835Aw2dGgj4tlRZheH1sgUf9vCYIGf8yNOjGsLHCAfK6JHIxiOT8qgBaytKDq3dF4LBOZu0bbL4kwfcApBQoYljV9WOLm5tzeBuAWDeeLnnhuXRxRqPVigrp6CKznPvjMkT55SisljnKsS9y7g8T0Lr1.UQjrRbRO2sbTFvRiLO3nuK5iQ30N7EfV1Npya9FxElen3youMRJSQgJ3n6hI4PU6Ts.ZB68TimRLux9s99XW8q2dv56kDZdgK8iA77G9KlE_vys6EsoZIYResTiczTGaEN3w9VWho04pbqdmFbjMX

With user agent:

$ curl -H "User-Agent: Mozilla/5.0 (Windows NT 11.0; Win64; x64)" https://www.iucr.org/__data/iucr/powder/QARR/col/cpd-1h.prn
   5.000       158
   5.020       174
   5.040       174
   5.060       146
   5.080       136
   5.100       163
   5.120       166
   5.140       174
   5.160       129
...

@clemisch
Add user agent to curl in QPA example
991e977 
Otherwise, download is blocked by Cloudflare
@sbillinge
Copy link
Contributor

@clemisch thanks for this. We are reworking some things about this repo on a branch called migration. Please could you do us a favor and see if the bug is fixed if you try and run this on the migration branch?

@clemisch
Copy link
Author

It is, because cpd-1h.prn is already included in the repo.

@sbillinge
Copy link
Contributor

Thanks @clemisch . We are preparing a pre-release of the migration branch for @vincefn to test. Once we are confident we didn't break anything we will merge it to into main and make a new release from there. If you are just trying to install pyobjcryst to run things, please feel free to install from migration. We hope to have this merged to main in the next few days which will make things easier for you.

@sbillinge
Copy link
Contributor

Thanks so much for your PRs btw. We really appreciate it.

@sbillinge
Copy link
Contributor

I will leave this open until we merge migration into main. If and when we decide to do that, this will be fixed and we can close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@clemisch @sbillinge