"Web Archive" redirects here. For the archiving service at web.archive.org, see Wayback Machine. For the Safari format of the same name, see webarchive. For the KDE format, see KDE WAR (file format).
Web archiving is the process of collecting portions of the World Wide Web and ensuring the collection is preserved in an archive, such as an archive site, for future researchers, historians, and the public. Due to the massive size of the Web, web archivists typically employ web crawlers for automated collection. The largest web archiving organization based on a crawling approach is the Internet Archive which strives to maintain an archive of the entire Web. National libraries, national archives and various consortia of organizations are also involved in archiving culturally important Web content. Commercial web archiving software and services are also available to organizations who need to archive their own web content for corporate heritage, regulatory, or legal purposes.
 Collecting the Web
 Methods of collection
 Remote harvesting
The most common web archiving technique uses web crawlers to automate the process of collecting web pages. Web crawlers typically view web pages in the same manner that users with a browser see the Web, and therefore provide a comparatively simple method of remotely harvesting web content. Examples of web crawlers frequently used for web archiving include:
There are numerous services that may be used to archive web resources "on-demand", using web crawling techniques. Those offering archives for use as legal evidence comply with Federal laws and standards.
 Database archiving
Database archiving refers to methods for archiving the underlying content of database-driven websites. It typically requires the extraction of the database content into a standard schema, often using XML. Once stored in that standard format, the archived content of multiple databases can then be made available using a single access system. This approach is exemplified by the DeepArc and Xinq tools developed by the Bibliothèque nationale de France and the National Library of Australia respectively. DeepArc enables the structure of a relational database to be mapped to an XML schema, and the content exported into an XML document. Xinq then allows that content to be delivered online. Although the original layout and behavior of the website cannot be preserved exactly, Xinq does allow the basic querying and retrieval functionality to be replicated.
 Transactional archiving
Transactional archiving is an event-driven approach, which collects the actual transactions which take place between a web server and a web browser. It is primarily used as a means of preserving evidence of the content which was actually viewed on a particular website, on a given date. This may be particularly important for organizations which need to comply with legal or regulatory requirements for disclosing and retaining information.
A transactional archiving system typically operates by intercepting every HTTP request to, and response from, the web server, filtering each response to eliminate duplicate content, and permanently storing the responses as bitstreams. A transactional archiving system requires the installation of software on the web server, and cannot therefore be used to collect content from a remote website.
Examples of commercial transactional archiving software include:
 Difficulties and limitations
Web archives which rely on web crawling as their primary means of collecting the Web are influenced by the difficulties of web crawling:
However, it is important to note that a native format web archive, i.e. a fully browsable web archive, with working links, media, etc., is only really possible using crawler technology.
The Web is so large that crawling a significant portion of it takes a large amount of technical resources. The Web is changing so fast that portions of a website may change before a crawler has even finished crawling it.
 General limitations
Not only must web archivists deal with the technical challenges of web archiving, they must also contend with intellectual property laws. Peter Lyman states that "although the Web is popularly regarded as a public domain resource, it is copyrighted; thus, archivists have no legal right to copy the Web". However national libraries in many countries do have a legal right to copy portions of the web under an extension of a legal deposit.
Some private non-profit web archives that are made publicly accessible like WebCite or the Internet Archive allow content owners to hide or remove archived content that they do not want the public to have access to. Other web archives are only accessible from certain locations or have regulated usage. WebCite cites a recent lawsuit against Google's caching, which Google won.
 Aspects of Web curation
Web curation, like any digital curation, entails:
Thus, besides the discussion on methods of collecting the Web, those of providing access, certification, and organizing must be included. There are a set of popular tools that addresses these curation steps:
A suite of tools for Web Curation by International Internet Preservation Consortium:
Other open source tools for manipulating web archives:
 See also
 External links
This article is based on one or more articles in Wikipedia, with modifications and additional content by SOURCES editors. This article is covered by a Creative Commons Attribution-Sharealike 3.0 License (CC-BY-SA) and the GNU Free Documentation License (GFDL). The remainder of the content of this website, except where otherwise indicated, is copyright SOURCES and may not be reproduced without written permission. (For information use the Contact form.)
SOURCES.COM is an online portal and directory for journalists, news media, researchers and anyone seeking experts, spokespersons, and reliable information resources. Use SOURCES.COM to find experts, media contacts, news releases, background information, scientists, officials, speakers, newsmakers, spokespeople, talk show guests, story ideas, research studies, databases, universities, associations and NGOs, businesses, government spokespeople. Indexing and search applications by Ulli Diemer and Chris DeFreitas.
For information about being included in SOURCES as a expert or spokesperson see the FAQ or use the online membership form. Check here for information about becoming an affiliate. For partnerships, content and applications, and domain name opportunities contact us.