Showing posts with label MediaWiki. Show all posts
Showing posts with label MediaWiki. Show all posts

Friday, October 16, 2009

Wikipedia uses the CLDR data

I travelled with Siebrand the other day and I learned that in order to provide plural support at translatewiki.net, he uses the information in the CLDR to know what languages need plural support and in what way.

The amazing thing was that for some languages the plural support in MediaWiki is different from the one indicated by the standard. There are also a number of languages where the CLDR did not have information about their plural support.

It is vital that the CLDR and MediaWiki agree on how to provide plural support for languages. The CLDR is the standard and should be complete and correct because it exists for any application.
Thanks,
      GerardM

Thursday, January 24, 2008

Localizing MediaWiki: a translator's perspective

The push is on to localize the core 500 MediaWiki messages for numerous languages. The language committee responsible for the creation of new Wikipedia projects has made the sensible decision that a language cannot have its own Wikipedia unless and until the core user interface is available in that language. As a big side benefit, MediaWiki is used to run thousands of other sites, so a single localization effort can catalyze all sorts of projects in a given language.

The Swahili Wikipedia has about 6500 articles, but only about 100 MediaWiki messages had been translated. The schizophrenic Swanglish interface produces gems like this: "Ficha logged-in users | Fichua my edits." It was beyond time to bring sw.wikipedia up to standard, so I decided to give it a go. In the interests of letting localizers for other languages know what they are getting into, here is a brief report on the process:

1) First, you need to get yourself approved on Betawiki, the site that oversees all the translations. You will need to create an account and join the language project. Thankfully, the people on the site are friendly, helpful, and fast.

2) You will need to download the files (unless you want to do all the work online, which is NOT recommended) and use a special translation assistance tool such as the free POedit or OmegaT. To download, go to the Translate page and select the following options:
  • I want to: Export translation in Gettext format

  • Group: MediaWiki messages (most used)

  • Language: [select your language]

  • Limit: 500 messages per page

Click "Fetch" and you will end up with a long, messy-looking document. Copy and paste that into a text editor like TextPad, save it in standard text format with a .po extension (the filename should be something like mylanguage.po), open it with your translation software, and voila!

Actually, not so fast. It took me a painful half hour or more to figure out that the file had to be saved in "UTF-8" rather than "Unicode," before I could actually open the file with POedit.

3) Let the games begin! At first, the translation goes very quickly, as you breeze through terms like "January" and "Comment." However, you soon start hitting challenging terms. The less of a computer presence your language already has, the more of a challenge the terms will be. "View source." "Full resolution." "Metadata." "Disambiguation pages." And long chunks like this:
This page is currently protected because it is included in the following {{PLURAL:$1|page, which has|pages, which have}} cascading protection turned on. You can change this page's protection level, but it will not affect the cascading protection.

Frustratingly, more than half of the translation strings do not have any accompanying explanations. You may have to click through Wikipedia special pages looking for an instance of the term, in order to figure out what is being talked about. Even a simple term like "block" that does not have an explanatory note becomes needlessly difficult; I used the word for "a block of text" until a later entry made it clear that the sense called for was "prevent access."

Messages that have code elements such as $1 (meaning that some text or number will be inserted in that position by the software) should especially have explanatory text, since the content of "$1" often makes a big difference in the words you use and the order you place your text and code elements. For example, "$1 logged-in users" could either render as "50 logged-in users," in which case the Swahili would be "Watumiaji $1 sasa," or "Show/hide logged-in users," in which case the Swahili should be "$1 watumiaji sasa."

The final frustration is that many of the messages - naturally, the ones you put off for last - are extremely long. One has to wonder if messages like this are crucial to establishing the core functionality of MediaWiki in any language:
Using the form below will rename a page, moving all\n
of its history to the new name.\n
The old title will become a redirect page to the new title.\n
Links to the old page title will not be changed; be sure to\n
check for double or broken redirects.\n
You are responsible for making sure that links continue to\n
point where they are supposed to go.\n
\n
Note that the page will '''not''' be moved if there is already\n
a page at the new title, unless it is empty or a redirect and has no\n
past edit history. This means that you can rename a page back to where\n
it was just renamed from if you make a mistake, and you cannot overwrite\n
an existing page.\n
\n
WARNING!\n
This can be a drastic and unexpected change for a popular page;\n
please be sure you understand the consequences of this before\n
proceeding.

The warning that should be posted is that the project will take a lot longer than you expect, and won't be nearly as straightforward as advertised.

Nonetheless, the draft Swahili translation is complete, after probably 16 hours of work. At the moment it is being reviewed in Kenya, and we will upload it as soon as we finish refining it. Meanwhile, a few observations are in order:

  • It really helps to have a good familiarity with computer terminology in general and Wikipedia in particular before starting to translate MediaWiki. If you don't know about "Watchlists" or "RSS Feeds," you will face challenges beyond your normal translation project.

  • You will benefit greatly from a working technical glossary and good dictionaries for your language. Fortunately, Swahili has had a number of successful localization projects (OpenOffice, Google, Microsoft Windows, and others), so I had a lot of resources to consult and an online Swahili dictionary that contains many IT terms and to which I could add new ones. This will not be the case for most other African or minority languages.

  • This project should not be done alone. Ideally you will have two people who are both knowledgeable about computers, both fluent in English and the project language, but one of whom is a native English speaker and one a native speaker of the language in question. Alternately, have someone on standby who can explain any problematic terms. I was lucky to work with Arthur Buliva, a Kenyan computer programmer, as we pitched translations back and forth over instant messenger.

  • A big challenge is that you cannot simply coin terms where none exist in your project language. Your users will need to understand the meanings behind the messages, usually without reference to a dictionary. If your language does not have a word for "subcategories" or "namespace" - well, welcome to the wonderful world of software localization!

I do not want to scare anyone away from trying to localize MediaWiki in their language, but I do want to paint a realistic picture of the task in front of you. It is not fast, and for most languages it is not easy, but the outcome - truly useful software in your language - will be its own reward.

Saturday, January 12, 2008

MediaWiki language development in 2008

MediaWiki is the software that runs Wikipedia. Within the Wikimedia Foundation it is used in many languages for many projects. Some of these projects, like the Bengali Wikipedia, are the biggest single source for a language on the Internet.

The BetaWiki project has become the hub of the MediaWiki localisation, not only is a Web interface provided, there is also support for "gettext" or ".po" files. This allows for the use of CAT Tools like OmegaT. Managing the localisation of several hundred languages is not easy and making sure that MediaWiki is properly localised is a struggle. We found that the quality of the language support is spotty. Policies in the WMF have changed requiring localisation as a precondition for a new project. Big improvements in the MediaWiki localisation have happened in the last few months.

In order to improve the MediaWiki usabilty we have received some funding from Hivos. This will make it possible to put a premium on the localisation of MediaWiki for languages not in Europe or North America.

Many of the languages that MediaWiki supports are different and we do get requests for changing the sort order. Typically we follow the CLDR but there are languages where the CLDR does not help us. It would be good if we knew how to work with the CLDR and share our work.

For American Sign Language, we have been told that for SignWriting an extension will be programmed allowing for a Wikipedia in sign languages. This we hope will stimulate the emancipation of all sign languages a lot. It will be a first to have encyclopedic content for sign languages.

NB today new Wikipedias are starting in Saterland Frisian, Crimean Tatar and Lower Sorbian. :) For many more languages preparations are being made in the Incubator. :)

Thanks,
Gerard

Friday, November 16, 2007

Localisation of MediaWiki

MediaWiki, the software that runs, among others Wikipedia, is localised in many languages. Most new projects get their localisation as part of the software, the only thing left after an installation is to customise the system messages where neede. When a language is truly new to MediaWiki, the localisation can be done locally or centrally. When the localisation is done centrally, the messages are merged almost daily into the MediaWiki core and become available in all Wikimedia Foundation projects within a few days and every four months in a new MediaWiki release.

So far all the new localisations happened as a result of a request for new Wikipedias. Yesterday I noticed that the localisation for the Tuvin language, was driven by a project that is not within the WMF. You then ask yourself if there are localisations for languages elsewhere. I asked Angela, and she pointed me to several constructed languages hosted by Wikia. Some of them, like Aeres and Anglish are so new, that the Wikia project is likely the first real effort to realise these languages. A private use code can be used to indicate the language used until there is sufficient maturity for recognition.

There are Wikipedias in over 250 languages. The quality of the localisation of MediaWiki however is still patchy. It has improved a lot with the great work of Niklas Laxström and because no new WMF project is started without a good localisation for the language.

With so many people involved, all volunteers, it is not easy to get and maintain the desired quality. This quality is however essential as MediaWiki is a tool that supports more languages then virtually any other application and has an installed base that is growing rapidly.

Thanks,
Gerard