Selenium print source code python

Here is a simple code all it does is it opens the url and gets the length page source and waits for five seconds and will get the length of page source again.

if __name__=="__main__":
    browser = webdriver.Chrome[]
    browser.get["//codefights.com/tournaments/Xph7eTJQssbXjDLzP/A"]
    initial = len[browser.page_source]
    print[initial]
    time.sleep[5]
    new_source = browser.page_source
    print[len[new_source]

see the output: 15722 48800

you see that the length of the page source increases after a wait? you must make sure that the page is fully loaded before getting the source. But this is not a proper implementation since it blindly waits.

Here is a nice way to do this, The browser will wait until the element of your choice is found. Timeout is set for 10 sec.

if __name__=="__main__":
    browser = webdriver.Chrome[]
    browser.get["//codefights.com/tournaments/Xph7eTJQssbXjDLzP/A"]
    try:
        WebDriverWait[browser, 10].until[EC.presence_of_element_located[[By.CSS_SELECTOR, '.CodeMirror > div:nth-child[1] > textarea:nth-child[1]']]]  # 10 seconds delay
        print["Result:"]
        print[len[browser.page_source]]
    except TimeoutException:
        print["Your exception message here!"]

The output: Result: 52195

Reference:

//stackoverflow.com/a/26567563/7642415

//selenium-python.readthedocs.io/locating-elements.html

Hold on! even that wont make any guarantees for getting full page source, since individual elements are loaded dynamically. If the browser finds the element it moves on. So make sure you find the proper element to make sure the page has been loaded fully.

P.S Mine is Python3 & webdriver is in my environment PATH. So my code needs to be modified a bit to make it work for Python 2.x versions. I guess only print statements are to be modified.

This article is a part of our Content Hub. For more in-depth resources, check out our content hub on Selenium Python Tutorial.

Retrieving the page source of a website under scrutiny is a day-to-day task for most test automation engineers. Analysis of the page source helps eliminate bugs identified during regular website UI testing, functional testing, or security testing drills. In an extensively complex application testing process, automation test scripts can be written in a way that if errors are detected in the program, then it automatically.

• saves that particular page’s source code.
• notifies the person responsible for the URL of the page.
• extracts the HTML source of a specific element or code-block and delegates it to responsible authorities if the error has occurred in one particular independent HTML WebElement or code block.

This is an easy way to trace, fix logical and syntactical errors in the front-end code. In this article, we first understand the terminologies involved and then explore how to get the page source in Selenium WebDriver using Python.

TABLE OF CONTENT

• What Is An HTML Page Source?
• What Is An HTML Web Element?
• How To Get Page Source In Selenium WebDriver Using Python?
• How To Retrieve HTML Source Of WebElement In Selenium?
• How To Retrieve JSON Data From An HTML Page Source In Python Selenium WebDriver?

What Is An HTML Page Source?

In non-technical terminology, it’s a set of instructions for browsers to display info on the screen in an aesthetic fashion. Browsers interpret these instructions in their own ways to create browser screens for the client-side. These are usually written using HyperText Markup Language [HTML], Cascading Style Sheets [CSS] & Javascript.

This entire set of HTML instructions that make a web page is called page source or HTML source, or simply source code. Website source code is a collection of source code from individual web pages.

Here’s an example of a Source Code for a basic page with a title, form, image & a submit button.

What Is An HTML Web Element?

The easiest way to describe an HTML web element would be, “any HTML tag that constitutes the HTML page source code is a web Element.” It could be an HTML code block, an independent HTML tag like
, a media object on the web page – image, audio, video, a JS function or even a JSON object wrapped within tags.

In the above example – is an HTML web element, so is and the children of body tags are HTML web elements too i.e.,

This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.

Here’s a short glimpse of the Selenium Python 101 certification from LambdaTest:

Get HTML Page Source Using driver.execute_javascript

In the previous example, we have to comment out [or replace] the “driver.page_source” line and add the following line. “driver.execute_script is a Selenium Python WebDriver API to execute JS in Selenium environment. Here, we execute a JS script that returns an HTML body element.

The output code looks like this-

As you can observe, it only returns the innerHTML of the body element. Like the last output, we do not get the whole page source. To get the entire document, we execute “document.documentElement.outerHTML”. The execute_script line now looks like this-

This gives us precisely the output we got using “driver.page_source.”

Fetch Page Source Using Python’s Request Library In Selenium WebDriver. This method has nothing to do with Selenium but you can check What Is Selenium?, it’s a purely ‘Pythonic’ way to get a webpage source. Here, we use Python’s request library to make a get request to the URL and save the request’s response, i.e., page source to an HTML file and print on the terminal.

Here is the script –

This method can be used to quickly store a webpage source code without loading the page in the Selenium-controlled browser. Similarly, we can use the urllib Python library to fetch the HTML page source.

Get HTML Page Source Using “view-source:” URL

This is rarely required, but you can append the target URL with “view-source” and load it in the browser window to load the source code and save it in manual testing.

Programmatically to take source code of screenshots in Python Selenium [if required], you can load the page using –

Get HTML Page Source In Selenium Python WebDriver Using XPath

The fourth method to make Selenium WebDriver get a page source is to use XPath for saving it. Here, instead of page_source or executing JavaScript we identify the source element, i.e., and extract it. Comment out the previous page source fetching logic and replace it with the following-

In the above script, we are using a driver method, “find_element_by_xpath,” to locate the web page’s HTML element. We enter the document using source node – "//*" and get its “outer HTML,” which is the document itself. The output looks the same as we got earlier using driver.page_source.

How To Find Broken Links Using Selenium WebDriver?

How To Retrieve HTML Source Of WebElement In Selenium?

To get the HTML source of a WebElement in Selenium WebDriver, we can use the get_attribute method of the Selenium Python WebDriver. First, we grab the HTML WebElement using driver element locator methods like [find_element_by_xpath or find_element_by_css_selector]. Next, we apply the get_attribute[] method on this grabbed element to get it’s HTML source.

Suppose, from pynishant.github.io, and we want to grab and print the source code of the div with id “div1”. The code for this looks like this-

Here’s the output –

Similarly, to get the children or innerHTML of a WebElement –

There is an alternative way of doing this and achieving same result –

How To Retrieve JSON Data From An HTML Page Source In Python Selenium WebDriver?

Modern applications are built with multiple APIs at play. And often, these API dynamically change the content of HTML elements. JSON objects have emerged as an alternative to XML response types. So, it has become essential for a pro Selenium python tester to handle JSON objects, especially those embedded in HTML tags. Python provides us with an in-built JSON library to experiment with JSON objects.

To demonstrate with an example – we load “//www.cntraveller.in/” in Selenium driver and look-out for SEO schema contained in to verify that logo URL is included in the “JSON” schema. By the way, if you feel confused, this “SEO schema” is useful to get web pages ranked on google. It has nothing to do with code-logic or testing. We’re using it just for demonstration.

We’ll be using LambdaTest for this demo-

The output contains logoURL and webElement source –

Code Breakdown

The following three lines import required libraries: Selenium WebDriver, Python’s JSON, and re library to handle JSON objects and use regular expressions.

Next, we configure our script for running it successfully on LambdaTest’s cloud, which is quite fast and smooth. It took me less than 30 seconds to get started [maybe because I had prior experience with the platform]. But even if you are a first-timer, it would take less than 1 minute. Register and login using Google and click on Profile to copy your username and access token.

We launch the driver in full-screen mode and load the cntraveller home page with the following line of code –

Now, we locate JSON objects containing script using XPath locator and delete the unnecessary semicolons to load the string in JSON format properly.

And then, we check if the logo URL is present. If present, we print it.

Also, we check if the telephone detail is present. If not, we print the source code of the WebElement.

Lastly, we quit the driver.

driver.quit[]

How To Get Page Source As XML In Selenium WebDriver?

If you’re loading an XML-rendered website, you may want to save the XML response. Here’s a working solution for making Selenium get XML page source –

Conclusion

You can use any of the above-demonstrated methods and leverage the agility & scalability of LambdaTest Selenium Grid cloud to automate your test processes. It lets you execute your test cases on 3000+ browsers, operating systems, and their versions. Also, you can integrate the automation testing flow with modern CI/CD tools and adhere to the best continuous testing practices.

So, start automating your day-to-day tasks and make your life easier with LambdaTest right away.

Happy Testing!

Nishant Choudhary

A Web Scraping Python Developer and Data Evangelist, Nishant also loves to evangelize startups and technologies by writing technical content.

How do I find Selenium source code?

How do you find the source of a page in Python?

How do I get the HTML source from Python?

How do I print visible text in Selenium?