========================
Pootle 2.8 release notes
========================

*Released on 16 August 2017*

Welcome to Pootle 2.8!

If you want to try it, check one of the following:

- :doc:`Installation instructions </server/installation>`
- :doc:`Upgrading instructions </server/upgrading>`


Changes in Requirements
=======================

- Django>=1.10.5,<1.11
- zlib1g-dev required for lxml
- MySQL support uses mysqlclient instead of MySQLdb
- django-transaction-hooks is no longer used


Major Changes
=============

- Performed a security audit.  Made various changes to close potential security
  vectors including DOS type atacks.
- Browse view optimisation. The performance of the brose mode now scales well
  for large projects.  Statistics are now in real time.
- Improved editor performance.  Retrieval of editable units now correctly
  retrieves pages and prevents massive database load.
- Timeline data.  The timeline data has been migrated and improved to store
  more data changes and clean up past data.  This is a step to providing more
  detailed change history and is needed to do unit level comparisons for Pootle
  FS.
- Statistics generation and display. We no longer use rqworkers to calculate
  statistics.  Instead these are live and always correct being updated on unit
  changes.  The system is faster then previous statistics calculations.
- Pootle FS (beta) - provides bidirectional version control synchronisation.


Details of changes
==================

Below we provide much more detail. These are by no means exhaustive, view the
`git log <https://github.com/translate/pootle/compare/2.7.6...2.8.0>`_ for
complete information.

- MySQL now uses the ``mysqlclient`` database driver instead of ``MySQLdb``,
  this is Django's :ref:`preferred database driver
  <django:mysql-db-api-drivers>`.
- Performed a security audit
- JavaScript fixes addressing performance and memory leaks in the editor
- Improved editor performance
- Extensive review and fixes to RTL layout
- Refactored code for syncing stores - as part of the internal rework of core
  functionality in Pootle we reworked synchronisation code.  This will
  eventually be dropped when we move fully onto Pootle FS.
- Changed the default ``robots.txt``. It is now a static file which you can
  adjust for your site. The site is ``allow`` by default.
- Editor:

  - Suggestions:

    - Support rejecting/accepting suggestions with comments

      - Suggesters are emailed based on new
        :setting:`POOTLE_EMAIL_FEEDBACK_ENABLED` setting

    - Accept suggestion without ``can review`` permissions if submitted
      translation is 100% match.
    - Users cannot submit suggestions that match a previously submitted
      suggestion or the current translation.

  - Current unit is shown at the top of the unit list with one context row,
    this makes it is easier to use the navigation bar and prevents UI
    jumpiness.
  - Faster terminology matching yielding more results by using stemming.
  - Special non-display characters are depicted by symbols from a custom-built
    font:

    - Properly display whitespace as special character e.g. non breaking space,
      newline, tab.
    - Technical details of escape sequences are omitted in the output displayed
      to end users, making it easier to enter correct data.

  - Buttons to add language specific special characters display helpful
    information in tooltips.
  - Errors fixed:

    - Fixed bugs in muting/unmuting checks
    - Fixed filtering translations by month
    - Fixed ``TypeError`` error when filter in editor gets no units
    - Fixed bug that prevented translators from clicking on context rows to
      edit those context units

  - Translation memory:

    - Display original and translations side-by-side for TM results
    - AmaGama Translation Memory is now queried using CORS
    - AmaGama is only queried if the source language is supported by amaGama

  - Perform all highlighting in the client.
  - Clearly present plurals to translator.
  - Allow text from the similar translations area to be selected.
  - Force word wrapping on long strings with no spaces.
  - Improvements on timeline.
  - Cross-language translation is now restricted to admins. It heavily impacts
    performance and translators are unlikely to require it while admins may
    have a valid reason to use it.
  - Removed ability to clear language or project dropdown to prevent
    performance degradation.  This prevents users inadvertantly hitting very
    expensive queries.
  - Alternate source language translations are no longer displayed for
    anonymous users to prevent performance costs for users who aren't able to
    translate.
  - Editor is disabled for users without the required permissions. Reviewing
    suggestions is similarly disabled without required permissions.
  - Check categories can now be used in dropdown to filter units. You can
    review all 'critical' check failures at once.
  - Auto-matched translations are now highlighted to indicate that they came
    from translation memory to prevent confusing users.
  - Incomplete plural translations may now be submitted.
  - String error reporting form is now clearer and prevents empty reports from
    being submitted.

- Added suggestion bulk management:

  - Provides filtering by user, etc.
  - Allows to review multiple suggestions at once and reject/accept them at
    once optionally providing a comment for the suggesters

- Configuration system - a generic system to store configuration information
  for Pootle.
- Plugin framework - allowing Pootle to use plugins to expand its
  functionality.
- Comment system
- Removed Plurr format checks
- Removed `ENChecker`
- Added support to have several formats in the same project
- Browse pages:

  - Refactored stats backend:

    - We now store statistics in the database and have removed the need for
      rqworkers to calculate stats.
    - The stats refresh notice has been removed as all stats are now
      up-to-date, always.
    - Faster stats retrieval is now possible as stats are always up-to-date and
      we can get it directly from the database.

  - Disabled items are hidden by default but admin users can select to shown
    them.
  - Changed order of columns to highlight latest activity and pending work:

    - Last updated data is now only shown to admin users

  - Altered order in which some items are listed by default:

    - Projects and languages are sorted by most recent translators changes to
      highlight activity
    - Virtual folders are sorted by priority to highlight most important
      strings to translate

  - Hid most of the special 'templates' language data as it is unnecessary and
    can be confusing.
  - Got rid of fat cookies:

    - Increases responsiveness and removes security issue
    - Most data is now stored in user session instead
    - Sidebar is no longer automatically open for anonymous users when an
      announcement changes.

  - Leaderboard on top panel and expanded stats panel:

    - The top panel will show the three users with the highest score.
    - Expanded stats shows the top contributors scores and other detailed
      information about current location.

  - Numbers are rendered in a locale aware fashion.
  - Improvements to the statistics table for overly long filenames and smaller
    screens.
  - Files dropdown no longer keeps references to empty directories.
  - Fixed issue where the "Back" button would sometimes not work.
  - Fixed issue with project dropdown when there are projects without a name.

- Search:

  - No longer autocompletes
  - Added pluggable search backend
  - Search widget is disabled if user cannot translate.  This is to prevent any
    load on the server for users who are not able to contribute.
  - Old 'Exact Match' was separated into 'Case-sensitive match' and
    'Phrase match' allowing finer-grained searches and removing the previous
    confusion about the actual intent of the options.

- Added team page:

  - Only for languages so far, and only available to language managers
  - Replaces permissions with roles
  - Provides direct access to suggestion bulk management

- Revamped user profile page. The aim is to slowly draw more information onto
  this page and make it a hub for translators.

- Removed for performance reasons:

  - Removed statistics from user profiles. Will be brought back in the future.
  - Removed export view. This has been replaced with TMX export functionality,
    download still remains.
  - Removed performance hogging "More stats" in admin dashboard. While it has
    some useful information there are better ways to get this data.

- Removed reports feature. This was a potential security area and data leak.
  We will bring this back now that we have finer grained change tracking.
- Pootle's own localization changes:

  - Updated translations. You can still `contribute translation updates for
    your language <http://pootle.locamotion.org/projects/pootle/>`_.
  - Now `compilejsi18n
    <http://django-statici18n.readthedocs.io/en/latest/commands.html#compilejsi18n>`_
    is used to compile JavaScript translations into assets, thus requiring
    `django-statici18n
    <http://django-statici18n.readthedocs.io/en/latest/index.html>`_ app.
  - Password reset email is now localizable in Pootle.
  - Multiple changes in localizable strings to ease translation.
  - Select2 localization is bundled to ensure Select2 is shown localized.

- Upload and download:

  - Disabled upload for non-PO projects as conflict handling currently only
    works in PO.
  - Admins can upload translations as other user allowing correct crediting for
    translations.
  - Fixed error for stores with no revision.
  - Added the ability to download TMX exports.

- New Machine Translation providers:

  - `Caighdeán <https://github.com/kscanne/caighdean/blob/master/API.md>`_ -
    Irish
  - `Welsh <http://techiaith.cymru/api/translation/?lang=en>`_

- Refactoring of models to increase performance, including dropping unnecessary
  indices.
- User input is sanitized for outgoing emails
- Usernames using latin1 characters are now allowed
- Improved RQ usage and new management commands
- Changed Pootle logo and styling
- Added the ability to use a custom logo with :setting:`POOTLE_CUSTOM_LOGO`
- Documentation updates


Pootle FS (beta)
----------------

:ref:`Pootle FS <pootle_fs>` enables synchronization of Pootle against a
filesystem, or version control system, handling conflict resolution and other
situations of two files being out of sync.

Pootle FS follows a git like command execution.  We've designed it such that we
expect there to be no data loss when conflicts are discovered. Any conflicts
are turned into suggestions which can be resolved in Pootle.

Pootle FS is still in beta as we'd like to make sure that all the bugs are
washed out before making it an official and default part of Pootle.

sync_stores and update_stores are still the default method of interacting with
Pootle.  We expect these to remain for some time, but expect the next version
of Pootle to use to Pootle FS infrastructure to manage and handle these
commands.

- Added admin UI to set up projects configuration and language mapping
- CLI - adds :djadmin:`info`, :djadmin:`fetch`, :djadmin:`resolve`,
  :djadmin:`sync`, :djadmin:`add` and :djadmin:`rm` commands
- LanguageMapper - allows differing codes on the filesystem vs Pootle
- FileMapper - maps the file layout on the filesystem to the expected Pootle
  layout
- Store de/serialization - makes it possible to customise and adapt file
  serialisation, most likely for slight deviations from the official file
  format specification.
- Removed the ability to add new TPs from the admin UI for Pootle FS projects,
  we will initialise new TPs differently in Pootle FS.



Development changes
-------------------

- Updated and pinned PyPI requirements:

  - From now on requirements will be pinned in order to simplify support and
    development.

- Tests:

  - Massive improvement in test framework.
  - Coverage increased from 55% to 94%.
  - Moved to tox.
  - Travis caching and optimisations.
  - Added JavaScript testing.

- Code sanity:

  - Python code cleanup/linting pep8/pyflakes/pep257 to increase code health.
  - Javascript code linting and cleanups.
  - CSS code linting and cleanups.

- Code polishing:

  - Moved all commands to argparse.
  - Moved shortcuts to Mousetrap.
  - JS improvements, move to React components.

- Triage meetings are now held on a weekly basis.


Command changes and additions
-----------------------------

- Running Pootle commands using :command:`manage.py` is no longer supported,
  use :command:`pootle` instead.
- :command:`pootle` command warns if configuration is missing.
- Changed commands:

  - :djadmin:`verify_user` and :djadmin:`purge_user` now accept multiple
    usernames.
  - :djadmin:`refresh_scores` now recalculates user scores and accepts
    multiple usernames. It can be run across projects and/or languages.
  - :djadmin:`contributors` command has been refactored in order to return more
    accurate results and has new options
    :option:`--since <contributors --since>`,
    :option:`--until <contributors --until>` and
    :option:`--mailmerge <contributors --mailmerge>`. The ``--from-revision``
    option has been removed.
  - :djadmin:`flush_cache` flushes ``default``, ``redis`` caches,
    accepts :option:`--rqdata <flush_cache --rqdata>`,
    :option:`--django-cache <flush_cache --django-cache>` options.
  - :djadmin:`export` is now able to export zipped TMX files per translation
    project with the :option:`--tmx <export --tmx>` option.
    :option:`--rotate <export --rotate>` option allows old files to be removed.
  - :djadmin:`init` now creates a development configuration with
    :option:`--dev <init --dev>` option.

- Added new commands:

  - :djadmin:`list_serializers` allows to view serializers and deserializers
    installed on your system.
  - :djadmin:`config` allows to get, set, list, append and clear configuration
    settings.
  - :djadmin:`init_fs_project`.
  - :djadmin:`set_filetype`.
  - :djadmin:`schema` allows to dump the database schema on MySQL which is
    useful for diagnosing differences in database schema.
  - :djadmin:`update_data` allows to update the stats data.

- Removed commands:

  - :command:`run_cherrypy`.
  - :command:`start` has been removed, use :djadmin:`runserver` instead.
  - :command:`refresh_stats`.
  - :command:`clear_stats`.


Changes in settings
-------------------

- Changes in settings:

  - MySQL database connections should now use ``STRICT_TRANS_TABLES``.
  - :setting:`POOTLE_TM_SERVER` no longer receives the ``MIN_SCORE`` parameter,
    as it was misleading and had questionable effects.
  - :setting:`POOTLE_TM_SERVER` now accepts a ``MIN_SIMILARITY`` parameter, to
    filter out results which might be irrelevant. To learn more, check the
    documentation on
    :setting:`MIN_SIMILARITY <POOTLE_TM_SERVER-MIN_SIMILARITY>`.
  - Changed the default value for `ACCOUNT_SESSION_REMEMBER
    <https://django-allauth.readthedocs.io/en/latest/configuration.html>`_ so
    now sessions are always remembered.
  - :setting:`POOTLE_MARKUP_FILTER` defaults to ``'markdown'``, and ``None``,
    ``'html'``, ``'textile'`` and ``'restructuredtext'`` values have been
    deprecated. Deployments using any deprecated markup must migrate manually
    to Markdown. This setting will be removed in the future since Markdown will
    be the only available markup.

- Added new settings:

  - :setting:`POOTLE_SCORES` accepts custom settings for user scores
    calculation.
  - :setting:`POOTLE_SEARCH_BACKEND` to allow configuring the search backend to
    be used.
  - :setting:`POOTLE_EMAIL_FEEDBACK_ENABLED` to allow disabling sending emails
    to suggesters when suggestions are accepted or rejected.
  - :setting:`POOTLE_CUSTOM_LOGO`, :setting:`POOTLE_FAVICONS_PATH`,
    :setting:`POOTLE_FS_WORKING_PATH` and :setting:`POOTLE_CANONICAL_URL`
    settings to allow easy customisations.
  - :setting:`POOTLE_SQL_MIGRATIONS`.
  - :setting:`AMAGAMA_SOURCE_LANGUAGES`.

- Removed settings:

  - ``POOTLE_QUALITY_CHECKER`` since the custom quality checkers feature is
    gone.
  - ``POOTLE_SCORE_COEFFICENTS`` has been removed and replaced with
    :setting:`POOTLE_SCORES`.


Credits
=======

This release was made possible by the following people:

Ryan Northey, Dwayne Bailey, Julen Ruiz Aizpuru, Taras Semenenko, Leandro
Regueiro, Igor Afanasyev, Claude Paroz, Safa Alfulaij, Rene Ladan, Kevin
Scannell, Jason P. Pickering, Eamonn Lawlor, Alexander Lakhin, Robbie Cole,
Rhoslyn Prys, Prasasto Adi, Nootan Ghimire, Mikhail Paulyshka, Mike Robinson,
leonardcj, Henrik Feldt, Francesc Ortiz, Allan Nordhøy, Christian Lohmaier,
Burhan Khalid, benbankes, Arash Mousavi, Andy Kittner, Adam Chainz.

And to all our bug finders, testers and translators, a Very BIG Thank You.
