Debian Codegraph @

README

raw
1 NAME
2 Scrappy - The All Powerful Web Spidering, Scraping, Creeping Crawling
3 Framework
4
5 VERSION
6 version 0.94112090
7
8 SYNOPSIS
9 #!/usr/bin/perl
10 use Scrappy;
11
12 my $scraper = Scrappy->new;
13
14 $scraper->crawl('http://search.cpan.org/recent',
15 '/recent' => {
16 '#cpansearch li a' => sub {
17 print $_[1]->{href}, "\n";
18 }
19 }
20 );
21
22 And now manually, ... without crawl, the above is similar to the
23 following ...
24
25 #!/usr/bin/perl
26 use Scrappy;
27
28 my $scraper = Scrappy->new;
29
30 if ($scraper->get($url)->page_loaded) {
31 $scraper->select('#cpansearch li a')->each(sub{
32 print shift->{href}, "\n";
33 });
34 }
35
36 DESCRIPTION
37 Scrappy is an easy (and hopefully fun) way of scraping, spidering,
38 and/or harvesting information from web pages, web services, and more.
39 Scrappy is a feature rich, flexible, intelligent web automation tool.
40
41 Scrappy (pronounced Scrap+Pee) == 'Scraper Happy' or 'Happy Scraper'; If
42 you like you may call it Scrapy (pronounced Scrape+Pee) although Python
43 has a web scraping framework by that name and this module is not a port
44 of that one.
45
46 FEATURES
47 Scrappy provides a framework containing all the tools neccessary to
48 create a simple yet powerful web scraper. At its core, Scrappy loads an
49 array of features for access control, event logging, session handling,
50 url matching, web request and response handling, proxy management, web
51 scraping, and downloading.
52
53 Futhermore, Scrappy provides a simple Moose-based plugin system that
54 allows Scrappy to be easily extended.
55
56 my $scraper = Scrappy->new;
57
58 $scraper->control; # Scrappy::Scraper::Control (access control)
59 $scraper->parser; # Scrappy::Scraper::Parser (web scraper)
60 $scraper->user_agent; # Scrappy::Scraper::UserAgent (user-agent tools)
61 $scraper->logger; # Scrappy::Logger (event logger)
62 $scraper->queue; # Scrappy::Queue (flow control for loops)
63 $scraper->session; # Scrappy::Session (session management)
64
65 Please see the METHODS section for a more in-depth look at all Scrappy
66 functionality.
67
68 ATTRIBUTES
69 The following is a list of object attributes available with every
70 Scrappy instance, attributes always return an instance of the class they
71 represent.
72
73 content
74 The content attribute holds the HTTP::Response object of the current
75 request. Returns undef if no page has been successfully fetched.
76
77 my $scraper = Scrappy->new;
78 $scraper->content;
79
80 control
81 The control attribute holds the Scrappy::Scraper::Control object which
82 is used the provide access conrtol to the scraper.
83
84 my $scraper = Scrappy->new;
85 $scraper->control;
86
87 ... $scraper->control->restrict('google.com');
88 ... $scraper->control->allow('cpan.org');
89 ... if $scraper->control->is_allowed($url);
90
91 debug
92 The debug attribute holds a boolean which controls whether event logs
93 are captured.
94
95 my $scraper = Scrappy->new;
96 $scraper->debug(1);
97
98 logger
99 The logger attribute holds the Scrappy::Logger object which is used to
100 provide event logging capabilities to the scraper.
101
102 my $scraper = Scrappy->new;
103 $scraper->logger;
104
105 parser
106 The parser attribute holds the Scrappy::Scraper::Parser object which is
107 used to scrape html data from the specified source material.
108
109 my $scraper = Scrappy->new;
110 $scraper->parser;
111
112 plugins
113 The plugins attribute holds the Scrappy::Plugin object which is an
114 interface used to load plugins.
115
116 my $scraper = Scrappy->new;
117 $scraper->plugins;
118
119 queue
120 The queue attribute holds the Scrappy::Queue object which is used to
121 provide flow-control for the standard loop approach to crawling.
122
123 my $scraper = Scrappy->new;
124 $scraper->queue;
125
126 session
127 The session attribute holds the Scrappy::Session object which is used to
128 provide session support and persistent data across executions.
129
130 my $scraper = Scrappy->new;
131 $scraper->session;
132
133 user_agent
134 The user_agent attribute holds the Scrappy::Scraper::UserAgent object
135 which is used to set and manipulate the user-agent header of the
136 scraper.
137
138 my $scraper = Scrappy->new;
139 $scraper->user_agent;
140
141 worker
142 The worker attribute holds the WWW::Mechanize object which is used
143 navigate web pages and provide request and response header information.
144
145 my $scraper = Scrappy->new;
146 $scraper->worker;
147
148 METHODS
149 back
150 The back method is the equivalent of hitting the "back" button in a
151 browser, it returns the previous page (response) and returns that URL,
152 it will not backtrack beyond the first request.
153
154 my $scraper = Scrappy->new;
155
156 $scraper->get(...);
157 ...
158 $scraper->get(...);
159 ...
160 my $last_url = $scraper->back;
161
162 cookies
163 The cookies method returns an HTTP::Cookie object. Note! Cookies can be
164 made persistent by enabling session-support. Session-support is enable
165 by simply specifying a file to be used.
166
167 my $scraper = Scrappy->new;
168
169 $scraper->session->write('session.yml'); # enable session support
170 $scraper->get(...);
171 my $cookies = $scraper->cookies;
172
173 crawl
174 The crawl method is very useful when it is desired to crawl an entire
175 website or at-least partially, it automates the tasks of creating a
176 queue, fetching and parsing html pages, and establishing simple
177 flow-control. See the SYNOPSIS for a simplified example, ... the
178 following is a more complex example.
179
180 my $scrappy = Scrappy->new;
181
182 $scrappy->crawl('http://search.cpan.org/recent',
183 '/recent' => {
184 '#cpansearch li a' => sub {
185 my ($self, $item) = @_;
186 # follow all recent modules from search.cpan.org
187 $self->queue->add($item->{href});
188 }
189 },
190 '/~:author/:name-:version/' => {
191 'body' => sub {
192 my ($self, $item, $args) = @_;
193
194 my $reviews = $self
195 ->select('.box table tr')->focus(3)->select('td.cell small a')
196 ->data->[0]->{text};
197
198 $reviews = $reviews =~ /\d+ Reviews/ ?
199 $reviews : '0 reviews';
200
201 print "found $args->{name} version $args->{version} ".
202 "[$reviews] by $args->{author}\n";
203 }
204 }
205 );
206
207 domain
208 The domain method returns the domain host of the current page. Local
209 pages, e.g. file:///this/that/the_other will return undef.
210
211 my $scraper = Scrappy->new;
212
213 $scraper->get('http://www.google.com');
214 print $scraper->domain; # print www.google.com
215
216 download
217 The download method is passed a URL, a Download Directory Path and a
218 optionally a File Path, then it will follow the link and store the
219 response contents into the specified file without leaving the current
220 page. Basically it downloads the contents of the request (especially
221 when the request pushes a file download). If a File Path is not
222 specified, Scrappy will attempt to name the file automatically resorting
223 to a random 6-charater string only if all else fails, then returns to
224 the originating page.
225
226 my $scaper = Scrappy->new;
227 my $requested_url = '...';
228
229 $scraper->download($requested_url, '/tmp');
230
231 # supply your own file name
232 $scraper->download($requested_url, '/tmp', 'somefile.txt');
233
234 dumper
235 The dumper method is a convenience feature that passes the passed-in
236 objects to Data::Dumper which in turn returns a stringified
237 representation of that object/data-structure.
238
239 my $scaper = Scrappy->new;
240 my $requested_url = '...';
241
242 $scraper->get($requested_url);
243
244 my $data = $scraper->select('//a[@href]')->data;
245
246 # print out the scraped data
247 print $scraper->dumper($data);
248
249 form
250 The form method is used to submit a form on the current page.
251
252 my $scraper = Scrappy->new;
253
254 $scraper->form(fields => {
255 username => 'mrmagoo',
256 password => 'foobarbaz'
257 });
258
259 # or more specifically, for pages with multiple forms
260
261 $scraper->form(form_name => 'login_form', fields => {
262 username => 'mrmagoo',
263 password => 'foobarbaz'
264 });
265
266 $scraper->form(form_number => 1, fields => {
267 username => 'mrmagoo',
268 password => 'foobarbaz'
269 });
270
271 get
272 The get method takes a URL or URI object, fetches a web page and returns
273 the Scrappy object.
274
275 my $scraper = Scrappy->new;
276
277 if ($scraper->get($new_url)->page_loaded) {
278 ...
279 }
280
281 # $self->content has the HTTP::Response object
282
283 log
284 The log method logs an event with the event logger.
285
286 my $scraper = Scrappy->new;
287
288 $scraper->debug(1); # unneccessary, on by default
289 $scraper->logger->verbose(1); # more detailed log
290
291 $scraper->log('error', 'Somthing bad happened');
292
293 ...
294
295 $scraper->log('info', 'Somthing happened');
296 $scraper->log('warn', 'Somthing strange happened');
297 $scraper->log('coolness', 'Somthing cool happened');
298
299 Note! Event logs are always recorded but never automatically written to
300 a file unless explicitly told to do so using the following:
301
302 $scraper->logger->write('log.yml');
303
304 page_content_type
305 The page_content_type method returns the content_type of the current
306 page.
307
308 my $scraper = Scrappy->new;
309 $scraper->get('http://www.google.com/');
310 print $scraper->page_content_type; # prints text/html
311
312 page_data
313 The page_data method returns the HTML content of the current page,
314 additionally this method when passed a string with HTML markup, updates
315 the content of the current page with that data and returns the modified
316 content.
317
318 my $scraper = Scrappy->new;
319 $scraper->get(...);
320 my $html = $scraper->page_data;
321
322 page_ishtml
323 The page_ishtml method returns true/false based on whether our content
324 is HTML, according to the HTTP headers.
325
326 my $scraper = Scrappy->new;
327
328 $scraper->get($requested_url);
329 if ($scraper->is_html) {
330 ...
331 }
332
333 page_loaded
334 The page_loaded method returns true/false based on whether the last
335 request was successful.
336
337 my $scraper = Scrappy->new;
338
339 $scraper->get($requested_url);
340 if ($scraper->page_loaded) {
341 ...
342 }
343
344 page_match
345 The page_match method checks the passed-in URL (or URL of the current
346 page if left empty) against the URL pattern (route) defined. If URL is a
347 match, it will return the parameters of that match much in the same way
348 a modern web application framework processes URL routes.
349
350 my $url = 'http://somesite.com/tags/awesomeness';
351
352 ...
353
354 my $scraper = Scrappy->new;
355
356 # match against the current page
357 my $this = $scraper->page_match('/tags/:tag');
358 if ($this) {
359 print $this->{'tag'};
360 # ... prints awesomeness
361 }
362
363 .. or ..
364
365 # match against a passed url
366 my $this = $scraper->page_match('/tags/:tag', $url, {
367 host => 'somesite.com'
368 });
369
370 if ($this) {
371 print "This is the ", $this->{tag}, " page";
372 # ... prints this is the awesomeness page
373 }
374
375 page_reload
376 The page_reload method acts like the refresh button in a browser, it
377 simply repeats the current request.
378
379 my $scraper = Scrappy->new;
380
381 $scraper->get(...);
382 ...
383 $scraper->reload;
384
385 page_status
386 The page_status method returns the 3-digit HTTP status code of the
387 response.
388
389 my $scraper = Scrappy->new;
390 $scraper->get(...);
391
392 if ($scraper->page_status == 200) {
393 ...
394 }
395
396 page_text
397 The page_text method returns a text representation of the last page
398 having all HTML markup stripped.
399
400 my $scraper = Scrappy->new;
401 $scraper->get(...);
402
403 my $text = $scraper->page_text;
404
405 page_title
406 The page_title method returns the content of the title tag if the
407 current page is HTML, otherwise returns undef.
408
409 my $scraper = Scrappy->new;
410 $scraper->get('http://www.google.com/');
411
412 my $title = $scraper->page_title;
413 print $title; # print Google
414
415 pause
416 This method sets breaks between your requests in an attempt to simulate
417 human interaction.
418
419 my $scraper = Scrappy->new;
420 $scraper->pause(20);
421
422 $scraper->get($request_1);
423 $scraper->get($request_2);
424 $scraper->get($request_3);
425
426 Given the above example, there will be a 20 sencond break between each
427 request made, get, post, request, etc., You can also specify a range to
428 have the pause method select from at random...
429
430 $scraper->pause(5,20);
431
432 $scraper->get($request_1);
433 $scraper->get($request_2);
434
435 # reset/turn it off
436 $scraper->pause(0);
437
438 print "I slept for ", ($scraper->pause), " seconds";
439
440 Note! The download method is exempt from any automatic pausing.
441
442 plugin
443 The plugin method allow you to load a plugin. Using the appropriate case
444 is recommended but not neccessary. See Scrappy::Plugin for more
445 information.
446
447 my $scraper = Scrappy->new;
448
449 $scraper->plugin('foo_bar'); # will load Scrappy::Plugin::FooBar
450 $scraper->plugin('foo-bar'); # will load Scrappy::Plugin::Foo::Bar
451 $scraper->plugin('Foo::Bar'); # will load Scrappy::Plugin::Foo::Bar
452
453 # more pratically
454 $scraper->plugin('whois', 'spammer_check');
455
456 ... somewhere in code
457
458 my $var = $scraper->plugin_method();
459
460 # example using core plugin Scrappy::Plugin::RandomProxy
461
462 my $s = Scrappy->new;
463
464 $s->plugin('random_proxy');
465 $s->use_random_proxy;
466
467 $s->get(...);
468
469 post
470 The post method takes a URL, a hashref of key/value pairs, and
471 optionally an array of key/value pairs, and posts that data to the
472 specified URL, then returns an HTTP::Response object.
473
474 my $scraper = Scrappy->new;
475
476 $scraper->post($requested_url, {
477 input_a => 'value_a',
478 input_b => 'value_b'
479 });
480
481 # w/additional headers
482 my %headers = ('Content-Type' => 'multipart/form-data');
483 $scraper->post($requested_url, {
484 input_a => 'value_a',
485 input_b => 'value_b'
486 }, %headers);
487
488 Note! The most common post headers for content-type are
489 application/x-www-form-urlencoded and multipart/form-data.
490
491 proxy
492 The proxy method will set the proxy for the next request to be tunneled
493 through.
494
495 my $scraper = Scrappy->new;
496
497 $scraper->proxy('http', 'http://proxy1.example.com:8000/');
498 $scraper->get($requested_url);
499
500 $scraper->proxy('http', 'ftp', 'http://proxy2.example.com:8000/');
501 $scraper->get($requested_url);
502
503 # best practice when using proxies
504
505 use Tiny::Try;
506
507 my $proxie = Scrappy->new;
508
509 $proxie->proxy('http', 'http://proxy.example.com:8000/');
510
511 try {
512 $proxie->get($requested_url);
513 } catch {
514 die "Proxy failed\n";
515 };
516
517 Note! When using a proxy to perform requests, be aware that if they fail
518 your program will die unless you wrap your code in an eval statement or
519 use a try/catch mechanism. In the example above we use Tiny::Try to trap
520 any errors that might occur when using proxy.
521
522 request_denied
523 The request_denied method is a simple shortcut to determine if the page
524 you requested got loaded or redirected. This method is very useful on
525 systems that require authentication and redirect if not authorized. This
526 function return boolean, 1 if the current page doesn't match the
527 requested page.
528
529 my $scraper = Scrappy->new;
530 $scraper->get($url_to_dashboard);
531
532 if ($scraper->request_denied) {
533 # do login, again
534 }
535 else {
536 # resume ...
537 }
538
539 response
540 The response method returns the HTTP::Repsonse object of the current
541 page.
542
543 my $scraper = Scrappy->new;
544 $scraper->get(...);
545 my $res = $scraper->response;
546
547 select
548 The select method takes XPATH or CSS selectors and returns a
549 Scrappy::Scraper::Parser object which contains the matching elements.
550
551 my $scraper = Scrappy->new;
552
553 # return a list of links
554 my $list = $scraper->select('#profile li a')->data; # see Scrappy::Scraper::Parser
555
556 foreach my $link (@{$list}) {
557 print $link->{href}, "\n";
558 }
559
560 # Zoom in on specific chunks of html code using the following ...
561 my $list = $scraper
562 ->select('#container table tr') # select all rows
563 ->focus(4) # focus on the 5th row
564 ->select('div div')->data;
565
566 # The code above selects the div > div inside of the 5th tr in #container table
567 # Access attributes html, text and other attributes as follows...
568
569 $element = $scraper->select('table')->data->[0];
570 $element->{html}; # HTML representation of the table
571 $element->{text}; # Table stripped of all HTML
572 $element->{cellpadding}; # cellpadding
573 $element->{height}; # ...
574
575 stash
576 The stash method sets a stash (shared) variable or returns a reference
577 to the entire stash object.
578
579 my $scraper = Scrappy->new;
580 $scraper->stash(age => 31);
581
582 print 'stash access works'
583 if $scraper->stash('age') == $scraper->stash->{age};
584
585 my @array = (1..20);
586 $scraper->stash(integers => [@array]);
587
588 store
589 The store method stores the contents of the current page into the
590 specified file. If the content-type does not begin with 'text', the
591 content is saved as binary data.
592
593 my $scraper = Scrappy->new;
594
595 $scraper->get($requested_url);
596 $scraper->store('/tmp/foo.html');
597
598 url
599 The url method returns the complete URL for the current page.
600
601 my $scraper = Scrappy->new;
602 $scraper->get('http://www.google.com/');
603 print $scraper->url; # prints http://www.google.com/
604
605 AUTHOR
606 Al Newkirk <[email protected]>
607
608 COPYRIGHT AND LICENSE
609 This software is copyright (c) 2010 by awncorp.
610
611 This is free software; you can redistribute it and/or modify it under
612 the same terms as the Perl 5 programming language system itself.
613