Custom formatters for numbers, dates etc.

[DeyLak]

Problem Description

I'm migrating my current project with no i18n library to Lingui and I've faced an issue, that I can't use a custom format for my numbers inside a message. I'm already relying on Intl APIs, but I have some custom functions with additional logic (like special handling of NaNs or some tweaks using DateTimeFormat.formatToParts) and tunned to my use cases.

Now I can't use them as custom formatters.

Proposed Solution

I want to have an opportunity to pass a custom format function into my macros calls (or even config it globally for the i18n instance).

I think, that can look smth like this in the code (modified docs example):

import { plural } from "@lingui/core/macro";
import { memoizedNumberFormat } from '@lingui/formatters'

const message = plural(numBooks, {
  one: plural(numArticles, {
    one: `1 book and 1 article`,
    other: `1 book and ${numArticles} articles`,
  }),
  other: plural(numArticles, {
    one: `${numBooks} books and 1 article`,
    other: `${numBooks} books and ${numArticles} articles`,
  }),
}, {
  format: {
    numBooks: num => num.toString(),
    numArticles: memoizedNumberFormat,
  }
  // or like this for simple cases
  // format: memoizedNumberFormat,
});

// ↓ ↓ ↓ ↓ ↓ ↓
// Generated message was wrapped for better readability

import { i18n } from "@lingui/core";
const message = i18n._(
  /*i18n*/ {
    id: "XnUh4j",
    message: `{numBooks, plural,
         one {{numArticles, plural,
            one {1 book and 1 article}
            other {1 book and {numArticles} articles}
         }}
         other {{numArticles, plural,
            one {{numBooks} books and 1 article}
            other {{numBooks} books and {numArticles} articles}
         }}
      }`,
    values: { numBooks, numArticles },
    format: {   
      numBooks: num => num.toString(),
      numArticles: memoizedNumberFormat,
    }
  }
);

Also may be add some default configuration options for i18n instance, like:

import { setupI18n } from "@lingui/core";
import { memoizedNumberFormat, memoizedDateFormat } from '@lingui/formatters'

const i18n = setupI18n({
  numberFormatter: memoizedNumberFormat,
  dateFormatter: memoizedDateFormat,
});

Alternatives Considered

For now, i've ended up providing 2 variables to my message instead of 1. It works, but that adds overhead for both translators and developers.

const formattedValue = formatInt(defaultValue);
const feedback = t`Default value: ${plural(defaultValue, {
        one: `${formattedValue} day`,
        other: `${formattedValue} days`,
      })}`

I've also though of how can I change my existing codebase to match the library formatting, but that way I don't see a way to handle my additional formatting logic and although Intl methods are the best we ever had for the dates and numbers localisation, they're still not always flexible enough to satisfy the design requirements.

Additional Context

I saw this issue https://github.com/lingui/js-lingui/issues/2265 about making intl formatters deprecated and I agree with the idea there, but I think it can't be done without a more general approach to how values formatting is handled inside the library.

Both i18n.number method and plural macro should have similar api and configuration opportunities. That way, we maybe can move the current Intl formatters implementations into a separate tree-shakable package for those, who need an easier adoption path.

I'm not sure, if I'm already have a good enough codebase knowledge for such a significant change. But we can discuss possible solutions and I can try making a PR for that with some assistance, if you agree with the idea itself.

[DeyLak]

Maybe another way to implement similar functionality is implementing ICU number skeletons which are supported by react-intl.

This solves the issue with different number formats inside messages and seems to be friendlier for translators, but lacks the flexibility of a custom formatter.

[timofei-iatsenko]

Something like you want could be already achieved with current code. I started investigating this part (because i haven't touched it for a long time), unfortunately there is a bit of mess and code/typings is not polished for that usecase.

Lingui supports ICU formatting as a first party, that means you can write such expression in the messages:

{value, number, percent}

Where percent is a style and by default it's a styles from Intl.NumberFormat constructor options

Additionally to these predefined styles you can pass your own formats to the i18n._ function:

i18n._(`{value, number, myStyle}`, {value: 55}, {formats: {myStyle: { /* Intl.NumberFormat options here */ }}})

• You can not define a custom styles globally on i18n instance now, you can only define this on each message. (this is easy to change though)
• You also could not provide a custom function as a formatter, only Intl options

However as you can see this pattern forces you to NOT use macro, and pass all values for placeholders manually, this is could be avoided using some lingui voodoo magic:

arg macro, was here for a while and it aimed to help writing custom icu expressions while keeping DX of the macro, the code with arg would look like that:

i18n._(msg`{${arg(value)}, number, myStyle}`, undefined, {formats: {myStyle: { /* Intl.NumberFormat options here */ }}})

arg macro basically prints placeholder name to the message without wrapping it in curly braces (as opposite to simple placing the variables inside the message) and store the value of the placeholder to the values field of the MessageDescriptor.

As you can see API for that case is not well ergonomical, unfortunately. You could not pass formats to the t macro neither msg macro directly, it has to be as a third parameter of i18n._ only. But i think this is also pretty easy to change.

Regarding your original usecase, i think using 2 variables in the message as you mentioned in the Alternatives Considered is good enough option. 1 variable is used to choose a correct plural case and second is formatted string.

[DeyLak]

@timofei-iatsenko thanks for the response!

Today after more digging into the library code, I've also found this possibility to pass custom formats to the i18n._ function. But the dx on that is pretty poor, as you mentioned.

That is also interesting, that I've discovered, that if I write a message like {value, number, {minimumFractionDigits: 3}} the defined format will be passed into the formatting function, but it doesn't work cause the message compilation throws.

Regarding that arg macro, I couldn't find, where from is it exported. It is neither in @lingui/core/macro nor in @lingui/react/macro or @lingui/core. I've also checked, that it is not defined here, though I see it in the JsMacroName enum.

But I don't see a reason to use or support it now, seems like the current documented macros are more powerfull and I think it will be a good dx and functionality improvement to support custom formats there.

So, maybe there is not too much work to achieve that custom formatting opportunity, as I thought. Now I see these tasks:

1. Add a global default formats to the setupI18n helper that should be merged with the ones passed into i18n._.
2. Add the opportinity to pass a function formatter to options.format. I'm not sure, what's the best way to do this, maybe modify the style helper to support not only objects and string, but also functions?

If you're agree with them, I can try making a PR. I think, that if we implement that, it will also be not that hard to add the support of whole number skeletons specification as an additional library plugin maybe.

Regarding the macro usage with custom formats, that seems trickier for me and I'm not sure what should be the right API there. Should they all accept options as a last parameter to be passed into the i18n._?

[timofei-iatsenko]

I think, that if we implement that, it will also be not that hard to add the support of whole number skeletons specification as an additional library plugin maybe.

Supporting number format skeletons should actually be quite straightforward. Lingui already supports date format skeletons—see this example: https://github.com/lingui/js-lingui/blob/68c29abf23974d8bffb6cadaacafc88e4760d3cb/packages/message-utils/src/compileMessage.ts#L45-L48

However, to use them, you need to have the arg macro; otherwise, there’s no way to pass placeholder values in macro style.

The arg macro has had an interesting history 😀. It was originally added by someone else, then removed by me because it wasn’t documented, and later re-added when I found a use case for it. But it still sits in the “voodoo magic” zone of Lingui—it mainly exists to support certain flows in the Vue integration, which is why it’s not exported in the core or React packages. That said, this would be easy to change.

Add a global default formats option to setupI18n that merges with any formats passed into i18n._.

This is a simple change, and I’m fine with it.

Allow passing a function formatter via options.format. I’m not sure what the best way to do this would be—maybe adjust the [style](https://github.com/lingui/js-lingui/blob/73f867c8af42d2759db3dd8c9d62e08e89ebad5d/packages/core/src/interpolate.ts#L23) helper to support not only objects and strings but also functions?

I don’t think this is a good idea, because it could open a Pandora’s box. If you allow formatters to be functions within the ICU expression itself, like:

{value, date, short}

…and then let someone replace short with an arbitrary function, it introduces the potential for inconsistency and misuse. For example, someone might format a date as a number instead.

I think a better solution would be to introduce entirely new ICU methods—and as far as I know (though I’d need to double-check the docs), this isn’t prohibited. So you could write something like:

{value, myFunction, myParameters}

…and then explicitly register myFunction. Do you see the distinction?

[DeyLak]

However, to use them, you need to have the arg macro; otherwise, there’s no way to pass placeholder values in macro style.

Ah, I see that now. I understood that without that macro, the interpolated value will be rendered inside the curly brackets making it impossible to pass custom formatting options in the message.

If I understand correctly (I'm not familiar with macros), to support arg macro, changes should be made in babel plugin traversing algorithm?

Also, I'm using SWC plugin, and as I see they have completely separate implementations due to the need of using Rust. I've seen the isArgDecorator and its usage in tokenizeNode function in the babel version. But I didn't find any mentions of it in the Rust version, except in some tests (probably copied from the TS implementation?)

I hope try looking into it more on the weekends, but I'd be grateful for some tips on where to look on the examples of how different macros are handled.

This is a simple change, and I’m fine with it.

Great, I'll look into it on the weekends and make a PR.

Do you see the distinction?

I get you, you're right, it was hasty of me to propose that solution. As it turns out, they're going to support the custom functions in the MessageFormat 2.0 which is still in development but recently hit its first stable release. I understand that it may be a lot of work and still early for the library to support it, cause the whole ecosystem needs to shift, but I'm looking forward to that change, cause it unlock a lot of cool stuff.

As for now, custom functions are not a part of the ICU spec, but they're not prohibited in the spec and seems that some implementations allow them. As I see that this string is valid {date, date, weekday=long;month=medium;day=short} I also tried writing something similar to what they're making in the MessageFormat 2.0: {date, date, weekday=long month=medium day=short} It parses as a valid AST in the AST explorer but it throws an error in the ICU playground. Seems that neither comas nor spaces are allowed in the current syntax.

Maybe we can make support for several arguments with our custom syntax or stick to the single argument as in messageformat v3?

[timofei-iatsenko]

If I understand correctly (I'm not familiar with macros), to support arg macro, changes should be made in babel plugin traversing algorithm?

Arg macro is already supported by babel plugin. It just lacks the typings in the @lingui/core/macro package and a proper documentation. So it's a very easy fix.

SWC version does not support arg macro, but this is also easy fix. So let's say it was not properly "released" previously, but now if you want to use it and it make sense for you it should be released (with SWC port + docs + typings)

As for now, custom functions are not a part of the ICU spec, but they're not prohibited in the spec and seems that some implementations allow them

Lingui uses @messageformat/parser package to parse icu strings. It supports custom functions parsing with parameters (i wrote this simple test to check):

  it("should compile custom function", () => {
    expect(compileMessage("{value, myFunc}")).toMatchInlineSnapshot(`
      [
        [
          value,
          myFunc,
        ],
      ]
    `)

    expect(compileMessage("{value, myFunc, myParam}")).toMatchInlineSnapshot(`
      [
        [
          value,
          myFunc,
          myParam,
        ],
      ]
    `)
  })

There is small issues with a transforming values on the lingui side (it takes only 1 argument now), easy fix.

On the runtime side this expressions just mapped to the appropriate functions and executed. Lingui could be extended to pass custom functions and support them in messages.

[DeyLak]

Sorry for the delay. I've created PR's for the global format options and for the arg macro support. I will try to look into custom functions support later.

[timofei-iatsenko]

Thanks for opening these PRs — I’m reviewing them now. Before we go too far, though, let’s take a step back and revisit the original issue and how these changes are intended to address it. I feel like we may have lost track of that a bit during the discussion.

[DeyLak]

My original request was to be able to use custom formatting functions inside the message syntax. Current PR's are not yet there, but I think they already solve some of my original use cases. I still think, I need to pass my own custom function as a formatter, specially for dates, but with custom formatting options I can cover most of my number formatting use cases.

After the discussion, it turned out, that currently there is an opportunity to use custom formats by passing Intl options. But it was not available with macro usage, cause the only way of passing the formats was as a third param of i18n._ call. So we agreed, that it is fine change to allow define the custom formats globally, so they can be used in macros as well. That is the first PR.

There is still no way to pass a custom format into the macro call itself, but I'm not sure, if this is really needed. Seems like global formats settings should be enough for most use cases.

However, to use the custom formatting inside macros, we discovered a need for the arg macro, that is simply a way to tell Lingui not to form ICU variable message syntax automatically (wrap a template expression in curly brackets) and leave it for user to define.

Babel plugin already had the arg macro code, but didn't have typings for it and docs. That is my second PR. SWC plugin didn't have the arg macro implementation, so the third PR implements it.

So, essentially these PRs are all needed to allow the use of user defined ICU message syntax pieces inside the macro calls alongside with traditional Lingui-style "you don't need to learn and use ICU". That includes also using the previously available default formats like percent. As I see it, this changes make a foundation for the next step which may be adding the support of custom formatting functions.

I think, that support for custom formatting functions is a useful addition to the library, especially in the sight of the fact that the need for them was acknowledged even by the authors of the MessageFormat 2.0.

As I see it, we came to an agreement, that custom ICU functions may be introduced in Lingui and be used like {value, myFunc, myFuncParam}.

Seems, like this custom functions should also be a configuration option for the setupI18n method. Maybe they can also be added into third options parameter of i18n._ call for consistency with formats.

I still see the issue with passing multiple parameters into the function. Maybe its better just to use the same approach as in custom formatters for messageformat v3, if we already use their library for message parsing.

After introducing the custom formatters, I see an opportunity to maybe move the current formatters implementations into a separate package as a next step. Seems it can simplify the code due to better segregation of resposibility. Maybe even drop support for them, like it was proposed in https://github.com/lingui/js-lingui/issues/2265. But i'm not sure that it is a good thing to completely remove the cached formatters implementations, cause they should be good enough for the most users.

[timofei-iatsenko]

Having custom formats or custom functions in the message itself the only make sense if you are going to change them between the translations. Saying other words you shifting the responsibility from source code to translation.

If you don't need this, why not just format this in the source code and pass already formatted value?

I asked for this summary because an original request was to pass a custom formatter into the plural number format function. But non if these changed will allow you to do so. You will still have to pass 2 variables, one to select a form and another one as formatted value.

[DeyLak]

My original concern was that it is more friendly and less error prone for the translator to use a single variable name in the plural macro call and in the plural cases. With current changes I can do that, cause I can do smth like:

plural(booksCount, {
  one: '{booksCount, number, myFormat} book',
  other: '# books',
})

or

plural(booksCount, {
  one: `{${arg(booksCount)}, number, myFormat} book`,
  other: '# books',
})

As far as I understand, the ICU syntax doesn't support nesting formatting inside the plural call like {var, plural, one {{#, number} book} ...}. So, you're right, if we want to allow the custom formatting of #, we should somehow pass a custom formatter inside the macro (and i18n._) call.

[timofei-iatsenko]

I just dug into the sourcecode one again, and it's possible to override default formatter for plural message in the current code:

  it("should use number format for # in the plural message", () => {
    const plural = prepare("{value, plural, one {} other {# and #}}")
    expect(
      plural(
        {
          value: 2,
        },
        { number: { style: "percent" } as Intl.NumberFormatOptions }
      )
    ).toEqual("200% and 200%")
  })