Using cURL with PHP

admin — Fri, 16 Oct 2009 23:42:33 +0000

Today, I’ll give you some hint about how to use cURL in PHP.

cURL is a library that allow a webserver to transfer files with a remote computer using a variety of Internet protocols. cURL stands for “Client URLs”, it is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, HTTP, HTTPS,GOPHER, LDAP, DICT, TELNET and FILE.

Once you’ve compiled PHP with cURL support, you can begin using the cURL functions.

The basic idea behind the cURL functions is that you initialize a cURL session using the curl_init(), then you can set all your options for the transfer via the curl_setopt(), then you can execute the session with the curl_exec() and then you finish off your session using the curl_close().

Using cURL from the command line is easy. Here is an example that uses PHP cURL functions to fetch the data from PowerOptions Website and prints it’s output. The data that I’m going to harvest would be the Company Name and Symbol.

Here is the function I’ve used:

public function scrape() {
$ch = curl_init();
 
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10");
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 60);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_URL, "http://www.poweropt.com/optionable.asp?fl=A");
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
curl_setopt($ch, CURLOPT_PROXY, "192.168.20.206:3128");
$html= curl_exec($ch);
if (!$html) {
echo "cURL error number:" .curl_errno($ch)."\n";
echo "cURL error:" . curl_error($ch)."\n";
}
 
curl_close($ch);
 
$dom = new DOMDocument();
@$dom-&gt;loadHTML($html);
$xpath = new DOMXPath($dom);
$nodeLists = $xpath-&gt;evaluate("//*[contains(text(), 'Company Name')]/ancestor::table[1]//tr");
for ($i = 1; $i &lt; $nodeLists-&gt;length; $i++) {
        $node = $nodeLists-&gt;item($i);
	$node = $node-&gt;firstChild;
	$tempNode = $node-&gt;firstChild;
	$name = $tempNode-&gt;nodeValue;
	$node = $node-&gt;nextSibling;
	$node = $node-&gt;nextSibling;
	$symbol = $node-&gt;nodeValue;
	//echo $name." | ". $symbol ."";
	//echo "".$name. "|". $symbol ."";
	//Saving the data autpmaticaly after scraping.
	mysql_query("INSERT INTO company (company_name, symbol) VALUES('$name','$symbol')");
	}
}

In the above example, curl_exec($ch) execute the returned data from the remote server and stored in the variable $html.

For more advance technique, I used PHP DOMDocument() Class for extracting the harvested data instead of using php preg_match_all function.

Source: php.net

My Scribbles » php curl example

Using cURL with PHP