Get Html Source Code From A Website And Then Get An Element From The Html File

I want to get HTML code of a website and then get a certain element from that HTML file. There are things that can get HTML code like ajax and jquery. I am using node and want it t

Solution 1:

For Node.js there are two native fetching modules: http and https. If you're looking to scrape with a Node.js application, then you should probably use https, get the page's html, parse it with an html parser, I'd recommend cheerio. Here's an example:

// native Node.js module
const https = require('https')
// don't forget to `npm install cheerio` to get the parser!
const cheerio = require('cheerio')

// custom fetch for Node.js
const fetch = (method, url, payload=undefined) => new Promise((resolve, reject) => {
    https.get(
        url,
        res => {
            const dataBuffers = []
            res.on('data', data => dataBuffers.push(data.toString('utf8')))
            res.on('end', () => resolve(dataBuffers.join('')))
        }
    ).on('error', reject)
})

const scrapeHtml = url => new Promise((resolve, reject) =>{
  fetch('GET', url)
  .then(html => {
    const cheerioPage = cheerio.load(html)
    // cheerioPage is now a loaded html parser with a similar interface to jQuery
    // FOR EXAMPLE, to find a table with the id productData, you would do this:
    const productTable = cheerioPage('table .productData')

    // then you would need to reload the element into cheerio again to
    // perform more jQuery like searches on it:
    const cheerioProductTable = cheerio.load(productTable)
    const productRows = cheerioProductTable('tr')

    // now we have a reference to every row in the table, the object
    // returned from a cheerio search is array-like, but native JS functions
    // such as .map don't work on it, so we need to do a manually calibrated loop:
    let i = 0
    let cheerioProdRow, prodRowText
    const productsTextData = []
    while(i < productRows.length) {
      cheerioProdRow = cheerio.load(productRows[i])
      prodRowText = cheerioProdRow.text().trim()
      productsTextData.push(prodRowText)
      i++
    }
    resolve(productsTextData)
  })
  .catch(reject)
})

scrapeHtml(/*URL TO SCRAPE HERE*/)
.then(data => {
  // expect the data returned to be an array of text from each 
  // row in the table from the html we loaded. Now we can do whatever
  // else you want with the scraped data. 
  console.log('data: ', data)
})
.catch(err => console.log('err: ', err)

Happy scraping!

Solution 2:

You will never be able to get the source code of a page that is not on the same domain as your page in javascript. Please take a look : http://en.wikipedia.org/wiki/Same_origin_policy

If the domain is the same you can get it by iFrames:

var url=".../xxxx"; // same domain !
var iframe=document.createElement("iframe");
iframe.onload=function()
{
var result= iframe.contentWindow.document.body.querySelector('.test').innerHTML);
alert(result);
}
iframe.src=url;
iframe.style.display="none";
document.body.appendChild(iframe);

var result access the iframe and select the element that have class named "test" and get its inner html in your case the result will be :

    #Some Stuff