Summary
-------
Cube for news storage and analysis



This cube provides an implementation of Semnews:

 - store news articles and tweets.

 - extract and synthetize information.

 - provide semantic useful and original visualisation.

 - analytics tools and datamining/machine learning processings.



Installation
------------

Creation of the instance:

 * Create an instance using: cubicweb-ctl create semnews <name-of-instance>

 * Create the instance's database using: cubicweb-ctl db-create <name-of-instance>


Add articles sources
--------------------

Source of articles could be created using:

 * Blogs/RSS feeds::

     session.create_entity('CWSource', name=<name of the source>, type=u'datafeed',
                           parser=u'rss-parser', lang=<lang of the source>,
                           url=<url of the blog/rss feed>,
                           config=u'synchronization-interval=120min')

 * Tweet::

     session.create_entity('CWSource', name=<name of the source>, type=u'datafeed',
                           parser=u'tweet-parser', lang=<lang of the source>,
                           url=<url of the blog/rss feed>,
                           config=u'synchronization-interval=120min')


The synchronization interval could be setted to a more specific value, or setted to "no" for manual synchronization
only.


Semnews comes with some predifined blogs/tweets/rss feeds:


 * Some french political blogs. You can add them using::

    cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_blogs_fr.py


 * Some international english newspapers. You can add them using::

    cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_newspapers.py


 * Some french newspapers. You can add them using::

    cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_newspapers_fr.py


 * Some french politician tweets. You can add them using::

    cubicweb-ctl shell <name-of-instance> <path-to-cube-code-source>/migration/examples_twitters_fr.py



Add Named Entities sources
--------------------------

Semnews is based on a named entities process, that you have to define::


     session.create_entity('NerProcess', name=<name of process>, host=<appid or sparql endpoint url>,
                           type=<rql or sparql>, lang=<optional lang of the ner source>,
                           request=<request to be performed>)

See the document of the NER cube for more details.
Example of source::

     session.create_entity('NerProcess', name=u'dbpedia38-en', host=u'ner',
                           type=u'rql', lang=u'en',
                           request=u'Any U WHERE X label %(token)s, X cwuri U, '
                                    'X ner_source NS, NS name "dbpedia38-en"')




Commands
--------

Semnews provide to commands:

 * A command to extract named entities from articles::

     cubicweb-ctl process-ner <name-of-instance>


 * A command to cleanup recognized entities according to some Dbpedia categories
   (see entities/external_resources.py)::

     cubicweb-ctl cleanup-ner <name-of-instance>
