License

Original content is copyright of Arutz 7.

Annotated corpus files, statistics and other information provided in this page are all copyright of Mila Knowledge Center

The information provided in this page is free. The Hebrew corpus, its resources and products are all licensed under the GNU Free Documentation License.

Background

The Corpus contains news and articles from Arutz 7 since 2001, which updates daily (until 2006). Text is available in HTML, plain ascii text, tokenized text in XML format. It is possible to obtain an XML version of the text morphologically annotated (with all possible analyses) and morphologically disambiguated (with the correct morphological analysis in context).

Every day, the front page of Arutz 7 is being scanned for updated news and articles and new material is being downloaded. The relevant text is being extracted from the downloaded pages, and then analyzed for document structure (paragraph, sentence and token segmentation). The texts are then being represented in XML according to the Hebrew Corpus schema.

The corpus contains 323,943 types.
The corpus contains 15,107,618 tokens.

Downloads

Researchers interested in obtaining access to the resources described in this page are kindly asked to fill the registration form

• Plain text - . Use cp1255 encoding in order to properly view the files.
• Hebrew Text Tokenized in XML format - Note that your browser will most probably render the XML files, so loading every file might take a while.
• Morphologically analyzed - Please send a request to Mila Knowledge Center.
• Morphologically disambiguated - Please send a request to Mila Knowledge Center.
• List of tokens by frequency - 500 most frequently used - a list of all tokens that appear in the corpus more than 10 times and their frequencies. Viewing is subject to GPL, no password is required.
• List of tokens by frequency - zip file - a list of all tokens that appear in the corpus more than 10 times and their frequencies. Viewing is subject to GPL, no password is required.
• List of bigrams by frequency - 1000 most frequently used bigrams - Viewing is subject to GPL, no password is required.
• List of all bigrams by frequency - zip file - Viewing is subject to GPL, no passwor d is required.

Other Corpuses which can be download from Mila site are:

The Medical Corpus

The Spoken Israeli Hebrew Corpus

The Knesset Protocols Corpus

The Economical Corpus - TheMarker Corpus

Hebrew With Nikud Corpus