How to fix 403 response when using HttpURLConnection in …
I was checking the active links in a website with selenium web driver and java. I have passed the links to the array and while verifying I am getting the response as 403 forbidden for all links in the site. It is just a public website anyone can access. The links are working properly when clicking manually. I wanted to know Why it is not showing 200 and what can be done on this situation.
This is for Selenium webdriver with Java
for(int j=0;j< ();j++) {
("Active Link address and status >>> ” + (j). getAttribute(“href”));
HttpURLConnection connection = (HttpURLConnection)new URL((j). getAttribute(“href”)). openConnection();
nnect();
String response = tResponseMessage();
int responsecode = tResponseCode();
connection. disconnect();
((j). getAttribute(“href”)+ “>>”+ response+ ” ” + responsecode);}
I expect the response code as 200, but the actual output is 403
asked Jun 19 ’19 at 13:26
Lipson T ALipson T A492 silver badges8 bronze badges
1
I believe your need to add the relevant Cookies to the HTTPUrlConnection, or even better consider switching to OkHttp library which is under the hood of Selenium Java Client
So you basically need to fetch the cookies from the browser using () function and generate a proper Cookie request header for the subsequent calls.
Example code:
(). getCookies(). forEach(cookie -> cookieBuilder
(tName())
(“=”)
(tValue())
(“;”));
OkHttpClient client = new OkHttpClient(). newBuilder()();
for (WebElement activelink: activelinks) {
Request request = new er()
(tAttribute(“href”)). addHeader(“Cookie”, String())
();
Response urlResponse = wCall(request). execute();
String response = ssage();
int responsecode = ();
(tAttribute(“href”) + “>>” + response + ” ” + responsecode);}
If you need nothing else but response code you can consider using HEAD method to avoid executing calls for the full URLs – this will allow you to save traffic and your test will be much faster.
answered Jun 19 ’19 at 14:17
Dmitri TDmitri T129k3 gold badges62 silver badges112 bronze badges
Had the same problem, user agent was the issue in my case (read more here:).
Also check what request methods are allowed on your website, you can do that by looking at any endpoint in “Network” tab in Chrome. It should list the allowed request methods, in my case I couldn’t use “HEAD”, but “GET” did the trick.
Code:
List
boolean brokenLink = false;
for (WebElement link: links) {
String url = tAttribute(“href”);
HttpURLConnection conn = (HttpURLConnection) new URL(url). openConnection();
tRequestMethod(“GET”);
tRequestProperty(“Content-Type”, “application/json”);
tRequestProperty(“User-Agent”,
“Mozilla/5. 0 (Windows NT 10. 0; Win64; x64) AppleWebKit/537. 36 (KHTML, like Gecko) Chrome/65. 0. 3325. 181 Safari/537. 36”);
int Code = tResponseCode();
if (Code >= 400) {
(“BROKEN LINK: ” + url + ” ” + Code);
brokenLink = true;
sertFalse(brokenLink);}
else {
(“Working link: ” + url + ” ” + Code);}}
answered Jan 21 at 11:30
KamilKamil112 bronze badges
Not the answer you’re looking for? Browse other questions tagged selenium selenium-webdriver response connection or ask your own question.
403 forbidden on selenium by visiting a page that works on …
I’m trying to visit this page using selenium. I’m using the same proxy as my regular browser, but while I can login without any issues on my chrome, selenium returns 403 forbidden and the proxy gets banned immediatelly after.
I’ve tried clearing my cache and cookies on my regular browser and it still works fine, while I cannot get selenium to visit the page without getting a 403. The proxies I’m using are perfectly working as they work for both requests and my regular Chrome. Is it possible that Selenium is getting somehow detected?
asked Dec 21 ’18 at 15:43
1
Easy ways to fix a 403 forbidden error message – Allconnect.com
At Allconnect, we work to present quality information with editorial integrity. While this post may contain offers from our partners, our opinions are our own. Here’s how we make it may seem intimidating at first, a ‘403 forbidden’ error message is easier to resolve than you might simply means that for some predetermined reason, the website’s content you’re trying to access is being reason might be within your control – but it’s more likely caused by something on the content-owner or server good news is there are a few quick and easy ways to fix the is a 403 forbidden error, anyway? If you ever had a treehouse as a kid, you may have posted a sign above the door that said “keep out. ” Essentially, that’s what a 403 forbidden error nically speaking, it’s an HTTP status code that means “access denied. ” You may also see it appear as:403 forbidden403 error403 forbidden accessBefore we jump into how to solve the issue, let’s explore why you’re getting the message in the first causes a 403 error? There are several possibilities as to why you’re seeing a 403. 1. The content is privateThe owner of the content has designated it as private. The reason? It could be anything from temporary, behind-the-scenes website updates to the website being subscriber-only access. 2. The content is user restrictedOnly authenticated users can access the content. This comes into play with organizations such as libraries and schools, both of which might have a limited number of content users. 3. The content is geographically restrictedSome sites only allow you access if you live in a specific geographic location. Examples of this are Netflix and Hulu. How does the website know where you’re located? Your IP address, which is unique to your computer and acts like a street address. 4. The IP address is prohibited or blockedThere can be a number of reasons for this but sometimes it boils down to spam or unwanted posts coming from a specific IP address. It can also be as simple as too many failed login to fix a 403 errorTake a few minutes to troubleshoot a 403 error. These suggested techniques aren’t complicated or overly technical and are well worth the time. We suggest that you try them in the order provided since you might fix the problem on the first attempt. Double-check the URLIt’s easy to mistype a URL so simply retyping may fix the 403 error. Clear your cache and cookiesYou’ve likely heard the terms cache and cookies but you should know that they play specific and different roles. Think of a cookie as leaving crumbs or tiny bits on information about where you’ve been on a site, what you like on the site, etc. Cache is more about speed since its purpose is to make loading time faster. An added benefit to clearing cache is that it may also improve your overall internet ’s how to clear cache and cookies on any browser. Keep in mind that clearing your cache and/or cookies may also clear your saved passwords. Double check your browser settings to be sure your passwords don’t get wiped out as well. Give it some timeVisit some other sites, get a cup of coffee or go for a walk. Allowing some time to go by may reveal that the 403 error was nothing more than in-progress website updates. Those updates are generally made as quickly as possible to minimize disruption. Contact the company, service or organization directlyIt could be that the 403 error is ongoing and that multiple people are experiencing it. You may need to reach out and let someone know about the issue. 5. Contact your internet service providerYour IP address may be blocked for some reason; however, this scenario is last on our list because it doesn’t happen often. If you’ve ruled out all of the above suggestions, contact your internet service provider to get some insight on whether or not the site is blocked. A 403 forbidden isn’t the most common error messageChances are, you’ve seen a 404 page not found error more often than a 403. Explore the rest of the Resource Center to learn more about common internet errors and what you can do about them. From cookies to caches: Understanding your browser history (and how to delete it) Sarah Harris — 4 min read What a 404 ‘page not found’ error is and ways to fix it Maria LeLaurin — 3 min read How the weather affects your internet Lisa Iscrupe — 2 min read Latest Saturday, October 2, 2021 Elon Musk’s Starlink is not the lone solution to the digital divide Ari Howard — 4 min read Monday, September 27, 2021 Trust in technology fell to an all-time low in 2021. Here’s why we’re losing faith. Joe Supan — 6 min read Saturday, September 25, 2021 What is a Comcast lift zone? Ari Howard — 2 min read
Frequently Asked Questions about selenium 403 forbidden
How do I fix 403 forbidden?
How to fix a 403 errorDouble-check the URL. It’s easy to mistype a URL so simply retyping may fix the 403 error.Clear your cache and cookies. … Give it some time. … Contact the company, service or organization directly. … Contact your internet service provider.Apr 22, 2019
Does 403 Forbidden mean?
What is a 403 Forbidden Error? The 403 Forbidden error is an HTTP status code which means that accessing the page or resource you were trying to reach is absolutely forbidden for some reason.Mar 5, 2020
Is Selenium safe to use Python?
Selenium can easily send standard commands of python to various browsers irrespective of all variation in the design of the browser. Python being a scripted language, there is no worrying about running a compiler for converting code from lines of the code to anything that can be implemented and utilized.Aug 26, 2019