Analytics : Ranking Texts By Visibility

This is a tip for maximizing the efficiency of translation for user facing content within your apps. In most applications, the 90/10 rule applies in that a small percentage of your user facing prompts (strings) account for the majority of views. Most companies use instrumentation to understand how users are interacting with their services, but typically don’t extend this to monitoring the relative visibility of user facing prompts.

This is worth doing because without that information, translators and localization managers don’t know which prompts are highly visible, and which ones are buried “beneath the fold”. As a general rule, higher visibility = higher impact, and higher risk.

We generally recommend that clients implement a fix forward workflow, where machine or AI translation is used as a placeholder and then reviewed and post-edited as needed by professional translators or copywriters.

With good instrumentation, you can focus their attention on the high priority/high visibility strings first, since those are more likely to be seen, before proceeding to the lower visibility strings. This ensures that if you have a capacity constraint on human oversight, which can happen if review is done by in-house or freelance language leads, the higher visibility strings are prioritized.

A good way to do this is to emit log events within the function that is called to display user facing strings. See the article below for a coding pattern to follow.

See: Future Proofing Your Codebase

Within this function, you will call another function to log this event. Since you are interested in relative, not absolute, views, you can use a randomization factor so this function is called one in X views, to avoid generating an excessive number of logging events that can degrade performance. Example below:

if random.randint(0,999) == 999:
  logEvent(stringKeyname, locale)

Then, you’ll need to write a small program or script that runs daily that performs the following tasks:

• Query the logging system or db for a list of text display events for the past N hours
• Group and count display events by keyname (filter out non-English locales so the results reflect the visibility of source language strings)
• Sort by view count from highest to lowest and normalize
• Assign each string a label such as P1, P2 and P3 based on its relative visibility

The query might look something like this:

SELECT * FROM StringEvents
  WHERE DATETIME > start_time_cutoff
    AND LOCALE = 'en-US'
  GROUP BY KEYNAME
  ORDER BY COUNT desc

Then do the following with this ranked list. Your TMS should provide an API that allows you to add and remove tags from strings. So for each string in this list, call the TMS API to apply the visibility tag to the string.

First you will want to normalize view counts on 0 to 1 or 0 to 100 scale so the counts reflect relative visibility. Then decide on what thresholds make sense for P1 (highest priority and oversight), P2 (medium priority and oversight) and P3 (everything else).

Then it is simply a matter of instructing your translation vendors and freelance reviewers to search for and review P1 strings, then P2 and then P3. This ensures you’ll get faster turnaround for the highest visibility strings, reducing the likelihood that an erroneous translation will be visible to users before it is corrected. And if there is a backlog in the post-edit or review steps, that will mostly affect lower visibility strings that are less likely to be noticed.

One caveat to be aware of is strings that are hidden behind a feature gate. Let’s say that you are rolling out a new appointment scheduling feature next month. These strings may not be visible to the instrumentation because customers can’t see them yet. This is where string keynames are handy. It is a good idea to have a pattern for keynames that goes something like this:

{appname}.{module|feature}.LoginPopoverDialog

Then translators can search for the module or feature name to look for strings that are tied to an important feature release. The string visibility instrumentation is mostly useful for gauging the relative visibility of strings post-release. This is especially useful when you add new interface languages, and want translators to focus on high visibility content first.

Ranking other asset types. This same methodology should be applied to other assets like web pages, help center articles, in app help blurbs, and so on. Here you can have an agent that runs daily, pulls relevant analytics and retags those assets in the TMS based on their calculated visibility level.

One especially useful tactic for long tail content like help center articles is to machine or AI translate them, and if they pop up about a certain visibility threshold, it kicks off a request for human review and post editing. This is a good way to optimize for speed, cost and quality, while bringing in human oversight for the content that is getting the most attention.

Conclusions

Most organizations use instrumentation throughout their stack, which should also be used to measure the visibility, and likely importance of user facing prompts, documents, and other assets, which can then be ranked in terms of priority in translation platforms.

Without this instrumentation, it is difficult for translators and reviewers to know which content needs more thorough review, and which can be passed through more automated and low cost workflows.

When properly implemented, this enables human translators and reviewers to prioritize the content that is most likely to be noticed by users before proceeding to lower priority content. This maximizes perceived quality, ensures that translation and review bottlenecks do not impact high visibility contact, and also allows for intelligent cost optimization and the safe use of fully automated translation workflows.

Related Reading

Global Ready Coding – a short list of low cost and no cost coding and design patterns that will prevent you from accruing tech debt that is expense and time consuming to retire.

Building For Runtime Localization – this white paper discusses how to build a dynamic translation pipeline that serves translations at runtime, and reduce the need for static message catalogs.

Choosing A Translation Management System – criteria you should consider in choosing a translation management system (TMS), written from a practical perspective for engineering and product teams.

Video Captioning and Subtitling – this article explains how to caption and subtitle streaming assets in multiple languages.