Journal of Computer Technology & Applications

Retrieval of Data from Hidden Web using Web Agent

  1. Das N. N. ✉
  2. Kumar E.

Abstract

AbstractThere is an increase in the number of data sources that can be queried across the World Wide Web (WWW). Such sources typically support HTML forms-based interfaces and search engines query collections of suitably indexed data. The data are displayed via a browser. As the Web is growing in recent years, huge amount of data are available under dynamic forms of publication, which are accessed by an HTML Form, known as legacy databases (known as hidden Web). To handle such types of situation fast generation of agents is required that can automatically fetch pages for further processing. In this paper, a method is explained for automatically generating agents to collect hidden Web pages (deep web pages). In this method a preexisting data repository is used for identifying the contents of these web pages. Number of experiments has been carried out with sites from different domains to show the accuracy of method.Keywords: Wrapper, hidden web, web agent, WWW, repository

References (21)

  1. Florescu D, Levy A, Mendelzon A. Database techniques for the World -Wide Web: A survey. SIGMOD Rec. (1998);27(3):59–74p.
  2. Lawrence S, Giles C. Searching the World-Wide Web. Science. 1998;280(4):98–100p.
  3. Bergman MK. The Deep Web: Surfacing Hidden V alue. : White Paper, Bright Planet; 2000.
  4. Embley DW, Campbell DM, Jiang YS, et al. Conceptual-model-based data extraction from multiple -record Web pages. Data Knowl Eng. 1999;31(3):227– 51p.
  5. Crescenzi V, Mecca G. Grammars have exceptions. Inf Sys. 1998;23(8):539–65p.
  6. Doorembos RB, Etzioni O, Weld D. A scalable comparison -shopping agent for the World-Wide Web. Proceedings of the First International Conference on Autonomous Agents ; Marina del Rey, USA. 1997. 39–48p.
  7. Liddle S, Embley D, Scott D, et al . Extracting data behind Web forms . Proceedings of the Workshop on Conceptual Modeling Approaches for e - Business. Tampere, Finland ; 2002. 38– 49p.
  8. Modica G, Gal A, Jamil HM. The use of machine-generated ontologies in dynamic information seeking . Proceedings of the 9th International Conference on Cooperative Information Systems ; Trento, Italy. 2001.433–48p.
  9. Raghavan S, Garcia -Molina H. Crawling the hidden Web . Proceedings of the 27th International Conference on Very Large Data Bases; Roma, Italy. 2001. 129–38p. Journal of Computer Technology & Applications Volume 5, Issue 2 ISSN: 2229-6964 (online), ISSN: 2347-7229 (print) JoCTA (2014) 26-34 © STM Journals 2014. All Rights Reserved Page 34
  10. Davulcu H, Freire J, Kifer M, et al . A layered architecture for querying dynamic Web content . Proceedings of the ACM SIGMOD International Conference on Management of Data ; Philadelphia, USA. 1999. 491–502p.
  11. He H, Meng WY, Lu YY, et al. Towards deeper understanding of the sear ch interfaces of the deep Web. World Wide Web. 2007;10:133−55p.
  12. Hong JL, Siew E, Egerton S. Information extraction for search engines using fast heuristic techniques. Data Knowl. Eng. 2010;69:169–96p.
  13. Hong JL, Siew E, Egerton S. WMS : Extracting multiple sections data records from search engine results pages. Proceedings of ACM SAC. 2010. 1696–701p.
  14. Barbosa L, Freire J. Searching for hidden - web databases. Proceedings of the 8th ACM SIGMOD International Workshop on Web and Databases; Baltimore, Maryland. 2005. 1–6p.
  15. Barbosa L, Freire J. Combining classifiers to identify online databases. Proceedings of the 16th International Conference on World Wide Web, WWW ’07; New York. 2007. 431–40p.
  16. Barbosa L, Freire J. An adaptive crawler for loc ating hidden -web entry points. Proceedings of the 16th International Conference on World Wide Web, WWW ’07. 2007;441–50p.
  17. Barbosa L, Nguyen H, Nguyen T, et al . Creating and exploring web form repositories. Proceedings of the 2010 International Conference on Management of Data, SIGMOD ’10 . New York: ACM; 2010. 1075–8p.
  18. Available from: http://daisen.cc.kyushu- u.ac.jp/TBDW/, 2009.
  19. Available at: http://www.w3.org/html/wg/ html5/, 2009.
  20. Zhao H, Meng W, Yu CT. Automatic extraction of dynamic record sections from search engine result pages. Proceedings of International Conference on Very Large Data Bases (VLDB) . 2006. 989–1000p.
  21. Gan G, Ma C, Wu J. Data clustering: Theory, algorithms, and applications. Soc Ind Appl Math. 2007.

Read more of it

Subscribe to one journal or build your own list of several. One invoice, one payment, access for a full year.

Subscribe to journals →

Publish your article

Peer-reviewed journals across every subject, with open access options. Send your manuscript to the right editor today.

Publish with us → Calculate your APC
Support