Android Web Crawler : Multithreaded Implementation

Web Crawler program automates task of indexing website pages.  It is also referred as Web Spidering and used extensively by Search Engines to provide effective and updated results for user search query.


Typical usage of Web Crawler includes, but not limited to:

  1. Page Indexing for Search Engines
  2. Page Content Scraping
  3. Dictionary Words Processing
  4. Syntax and Structure Validation

Given a root url, web crawler crawls content of current page and add urls extracted in processing queue of uncrawled urls. Once a page is crawled, data of crawled page is stored in database for later processing as per requirement. Task is time consuming if hyperlinks are crawled sequentially. We will create android web crawler example application which will perform parallel execution of crawling task. SQLite database will be used for saving crawled url’s record.

Android Web Crawler Example Application

 

[su_button url=”https://github.com/androidsrc/WebCrawlerUsingJsoup” target=”blank” style=”stroked” background=”#51d461" color=”#ffffff” size=”6" center=”yes” radius=”0" icon=”icon: arrow-circle-o-down”]Download Full Source Code[/su_b[/su_button]p> 

[su_yo[su_youtube url=”https://www.youtube.com/watch?v=W4FxInOcUME”]p>1. Create an android application

2. Preparing Application Manifest File

Add permission for using Internet as crawling requires processing URL with HttpURLConnection. Final AndroidManifest.xml will be as below.

You may also like...

2 Responses

  1. Atif Afridi says:

    sorry but this is not working with it crashes in debug mode also

    @Override
    protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);

    crawlingInfo = (LinearLayout) findViewById(R.id.crawlingInfo);
    startButton = (Button) findViewById(R.id.start);
    urlInputView = (EditText) findViewById(R.id.webUrl);
    progressText = (TextView) findViewById(R.id.progressText);

    crawler = new WebCrawler(this, mCallback);
    }
    // onCreate ENDS
    please help me

Leave a Reply

Your email address will not be published. Required fields are marked *