Question: how do I write & run a new profile? Confusion points in quick start guide

[arrowtype]

I am experimenting with writing a new custom profile, so I will record parts of the experience here, in hopes to help other people to get through the same thing, and ultimately in hopes that the docs can be improved a bit for other beginners.

[This issue is a work in progress and will be edited a few times as I work on this. Feel free to comment if you like, before it is done, however.]

Phase one: making fontbakery command recognize new profile

My beginner problems and solution (Click to expand)

I got stuck at trying to run what I have made so far (basically just the "hello world" example at this stage).

So far, I have tried to follow the instructions at https://font-bakery.readthedocs.io/en/stable/developer/writing-profiles.html.

I have copied from this and modified slightly to have different naming. My current progress is here: https://github.com/typotheque/fontbakery/blob/4d6c10adb591331c1cde457326beb5f8920434e6/Lib/fontbakery/profiles/typotheque.py

Then, I tried to run the following:

fontbakery check-profile typotheque /path/to/fonts

I also tried making a specific check-typotheque command, with this code.

▶ fontbakery check-typotheque /Users/stephennixon/type-repos/tptq/fontbakery-testing/tremoloVF     
usage: fontbakery [-h] [--list-subcommands] [--version]
                  [{build-contributors,check-adobefonts,check-fontval,check-googlefonts,check-opentype,check-profile,check-ufo-sources,check-universal,generate-glyphdata}]
fontbakery: error: argument subcommand: invalid choice: 'check-typotheque' (choose from 'build-contributors', 'check-adobefonts', 'check-fontval', 'check-googlefonts', 'check-opentype', 'check-profile', 'check-ufo-sources', 'check-universal', 'generate-glyphdata')

I then made a fontbakery-venv as recommended in the docs, and installed the modified fontbakery files with pip install -e . This didn’t seem to work quite as I needed it to at first, but I used which fontbakery to find that I wasn’t calling the right thing. I had to basically restart my venv and reinstall to make it work. This isn’t the exact order I did things in, but it’s close:

(fontbakery-venv) 
▶ which fontbakery
/Library/Frameworks/Python.framework/Versions/3.7/bin/fontbakery

(fontbakery-venv) 
type-repos/tptq/fontbakery-tptq  tptq ✗                                                                                   24m ⚑  
▶ deactivate 

type-repos/tptq/fontbakery-tptq  tptq ✗                                                                                   24m ⚑  
▶ source fontbakery-venv/bin/activate

(fontbakery-venv) 
type-repos/tptq/fontbakery-tptq  tptq ✗                                                                                   23m ⚑  
▶ pip install -e .  

(fontbakery-venv) 
type-repos/tptq/fontbakery-tptq  tptq ✗                                                                                   24m ⚑  
▶ which fontbakery                   
/Users/stephennixon/type-repos/tptq/fontbakery-tptq/fontbakery-venv/bin/fontbakery

I think I also had to add the following to the profile to make it work:

from fontbakery.checkrunner import Section
profile = profile_factory(default_section=Section("Typotheque"))

I also found that to run a specific new profile with the check-profile command, you must pass its file path, like so:

▶ fontbakery check-profile Lib/fontbakery/profiles/typotheque.py path/to/fonts/*.ttf

At this stage, the profile looks like this.

Potential improvements for docs

These ideas reflect my personal experience/opinion, so please take these ideas with a grain of salt.

• The “Writing Profiles” guide could start with more actionable, basic example with step-by-step instructions. For example:
  • It could also include information about pip installation
  • And then it could give just a hello world example, which someone could copy-paste
  • Maybe it could also link to a hello world in an example repo
  • If from fontbakery.checkrunner import Section \n profile = profile_factory(default_section=Section("Typotheque")) must be added, please include that in the hello world example
  • It should probably include instructions on how to make a new command
  • Then, it could give more information about the optional parts of a profile. Currently, the page starts with a few seemingly in-depth features that are hard to understand as a new reader.
• If the contributor guide recommends a venv called fontbakery-venv, I think this should be included in the .gitignore, correct? (It currently isn’t.)
• The first command of "Running Profiles" seems to maybe be incorrect. Doesn’t it need to give the relative filepath to the profile?

Phase two: "Profile fails expected checks test"

I have now gotten fontbakery to recognize the new profile. However, it is giving the error Profile fails expected checks test. Details inside this dropdown:

registering expected checks (Click to expand)

At this stage, the profile looks like this.

This was the error from it:

type-repos/tptq/fontbakery-tptq  tptq ✗                                                                                  31m ⚑  ⍉
▶ fontbakery check-profile Lib/fontbakery/profiles/typotheque.py /Users/stephennixon/type-repos/tptq/fontbakery-testing/tremoloVF
Traceback (most recent call last):
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/fontbakery-venv/bin/fontbakery", line 11, in <module>
    load_entry_point('fontbakery', 'console_scripts', 'fontbakery')()
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/cli.py", line 22, in main
    "fontbakery.commands." + subcommand_module, run_name='__main__')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 208, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/commands/check_profile.py", line 346, in <module>
    sys.exit(main())
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/commands/check_profile.py", line 229, in main
    profile = get_profile()
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/commands/check_profile.py", line 214, in get_profile
    imported = get_module(args.profile)
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/commands/check_profile.py", line 195, in get_module
    imported = get_module_from_file(name)
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/commands/check_profile.py", line 187, in get_module_from_file
    profile.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "Lib/fontbakery/profiles/typotheque.py", line 117, in <module>
    profile.test_expected_checks(expected_check_ids, exclusive=True)
  File "/Users/stephennixon/type-repos/tptq/fontbakery-tptq/Lib/fontbakery/checkrunner.py", line 1109, in test_expected_checks
    '\n'.join(message)))
fontbakery.checkrunner.SetupError: Profile fails expected checks test:
missing checks: com.typotheque/examples/hello, com.typotheque/examples/ttf_has_glyphs;
unexpected checks: com.google.fonts/check/family/panose_familytype, com.google.fonts/check/family_naming_recommendations, 

# ETC. 64 more check IDs omitted 

I am confused by this ... I have the following in my code near the top of the profile:

expected_check_ids = (
    'com.typotheque/examples/hello',
    'com.typotheque/examples/ttf_has_glyphs',
)

and both of those are in the code, as shown in the Quick Start but named for this profile. However, the error is calling those checks as missing checks.

...my profile also has the final line of:

profile.test_expected_checks(expected_check_ids, exclusive=True)

Have I missed something here?

UDPATE

I have found that I can get the check running with the following code, where I exclude the profile_imports = ('fontbakery.profiles.universal',) temporarily. This works and even allows me to run the ghmarkdown option.

🍞 PASS: Simple "Hello World" example.

• [com.typotheque/check/hello](https://font-bakery.readthedocs.io/en/latest/fontbakery/profiles/<Section: Typotheque>.html#com.typotheque/check/hello)

• 🍞 PASS Hello World

However, I still need to find how to import other checks.

UPDATE 2

I have solved this expected checks by taking some cues from the googlefonts profile. This included a couple of additions:

At the top:

from fontbakery.profiles.universal import UNIVERSAL_PROFILE_CHECKS

Near the top:

TYPOTHEQUE_CHECK_IDS = [
    'com.typotheque/check/hello',
]

EXPECTED_CHECK_IDS = \
    UNIVERSAL_PROFILE_CHECKS + \
    TYPOTHEQUE_CHECK_IDS

Last line:

profile.test_expected_checks(EXPECTED_CHECK_IDS, exclusive=True)

So, I think this is actually closer to a "Quick Start" / "Hello World" example, though obviously others would have to swap in their own foundry naming.

Suggestions for docs

• Maybe update the "Hello World" with a complete example, similar to what I linked to just above?
• Please explain why it is necessary to use both of the following (or if there is a better way?)
  • from fontbakery.profiles.universal import UNIVERSAL_PROFILE_CHECKS
  • profile_imports = ('fontbakery.profiles.universal',)

[kontur]

I think I also had to add the following to the profile to make it work:

from fontbakery.checkrunner import Section profile = profile_factory(default_section=Section("Typotheque"))

I second that from when I was writing a custom profile a while back. It's not super clear but seems to be required, even if that is your "only" section.

Personally also the profile_imports syntax is not super obvious (basically, python modules/methods). I recall it took me a while to understand how to pick only specific tests of a profile, or how to link to local relative files with checks or conditions.

[arrowtype]

Hmm, yeah I bet I will also need to experiment to get that right. Do you happen to have an open example of where you did pull in a specific check from another profile?

[kontur]

This includes a few different examples:

profile_imports = [
    [
        # Only a specific condition from the fontbakery module
        "fontbakery.profiles.shared_conditions", ["is_cff"]
    ],
    [
        # Only a specific condition from a local relative module (conditions.py)
        ".conditions", ["psautohint_stats"]
    ],
    [
        # A bunch of fontbakery profiles with all their checks
        "fontbakery.profiles",
        [
            "cff",
            "cmap",
            ...
            "post",
            "shared_conditions"
        ]
     ],
    [
        # One specific check from a profile in the fontbakery module
        "fontbakery.profiles.googlefonts", [
            "com_google_fonts_check_hinting_impact"
        ]
    ]
]

I'm not sure does it work so that omitting the second list includes all checks of a module, but at least if you want to cherry pick yourself some checks this works.

[graphicore]

There's a lot going on here, I'll try to answer some things.

First: there are separate options how to run a profile with Font Bakery. In general if you want to make a custom profile, I wouldn't implement it in a clone/fork of Font Bakery. Just use the file path to the file that contains the profile OR make a module and install it into your virtual environment.

You should only work in a clone of Font Bakery if you either want to contribute or if you want to fork it and hence also maintain the fork. The "Source Code Contributor Guide" is meant for contributing.

The “Writing Profiles” guide could start with more actionable, basic example with step-by-step instructions. […]

We could make add a "Step by Step Beginners Guide to writing profiles"

It should probably include instructions on how to make a new command

This is IMO not a direction authors of custom profiles should go. Maybe just write a one line shell command next to your custom profile or even in your users ~/.bin

Then, it could give more information about the optional parts of a profile. Currently, the page starts with a few seemingly in-depth features that are hard to understand as a new reader.

Yes, it's more like documentation, less like a guide, which is different, but a simpler guide could be present somewhere. Also, maybe you should have tried to read it again, I believe the stuff you did not understand is all in the very beginning of the document, with some exceptions, so maybe it's just necessary to put it there. I'd call it rather documentation than a guide, because the way one reads documentation is not linear, it's more like jumping on the page between passages and information.

I think one source of confusion was to use the "contributing" document along with the "writing profiles" document. So clearly, there's a need to improve the conversion funnel for the user.

If from fontbakery.checkrunner import Section \n profile = profile_factory(default_section=Section("Typotheque")) must be added, please include that in the hello world example

This is documented in detail at the beginning of the documentation: From automatic discovery to full control

I second that from when I was writing a custom profile a while back. It's not super clear but seems to be required, even if that is your "only" section.

No, it is not required.

I can copy and paste the example given in 'writing profiles' into a file, then run:

(venv3) commander@draugr:~/Projects/googlefonts> fontbakery  check-profile ./fontbakery-example-profile/my-custom-profile.py ~/Desktop/NotoSans-Regular.ttf

# Note, the profile is not in the fontbakery project directory:
(venv3) commander@draugr:~/Projects/googlefonts> which fontbakery
/home/commander/Projects/googlefonts/fontbakery/venv3/bin/fontbakery

The first command of "Running Profiles" seems to maybe be incorrect. Doesn’t it need to give the relative filepath to the profile?

Depends if your profile is installed as a module or not. I'm not exactly sure what you mean actually, but the one example uses a relative path and the other a module name, just look at the extension .py if you are not sure what is what, but there are also comments above (under "execute" in Quick start). However, you can also use an absolute path, FWIW.

(venv3) commander@draugr:~/Projects/googlefonts> fontbakery  check-profile /home/commander/Projects/googlefonts/fontbakery-example-profile/my-custom-profile.py ~/Desktop/NotoSans-Regular.ttf

About

[…] "Profile fails expected checks test"

• Maybe update the "Hello World" with a complete example, similar to what I linked to just above?

I believe it's mostly complete, but maybe we need a more verbose guide. But who's going to read it? If it's wrong or missing things we should correct that.

Please explain why it is necessary to use both of the following (or if there is a better way?) from fontbakery.profiles.universal import UNIVERSAL_PROFILE_CHECKS profile_imports = ('fontbakery.profiles.universal',)

There's a lot of info about what profile_imports does and why it's there, here's a shorter one, but there's a whole section.

profile_imports can be used to mix other profiles into this profile

if you follow the UNIVERSAL_PROFILE_CHECKS variable you'll see it ends up in EXPECTED_CHECK_IDS and that ends up in profile.test_expected_checks(EXPECTED_CHECK_IDS, exclusive=True)

Now profile.test_expected_checks is documented this describes what it is good for. Please also search for this sentence:

Since we have the profile reference we can also use the expected_check_ids self check method.

This is optional, hence we can (not must) use it. It's meant as self-check, similar to a check-sum. If you want to use it, you need a reference to a profile and that's why you create it: profile = profile_factory(default_section=Section("Typotheque")).

Importing UNIVERSAL_PROFILE_CHECKS in a way defeats the self-check, because you don't have control over the check id's in that list, you could maintain the list yourself in your custom profile, to be sure you don't miss it when a profile that is a dependency of your profile of you changes which checks it defines. But also, if you don't get why to use profile.test_expected_checks or if you don't need that kind of (self-)control, you should probably not use it in the first place.

[arrowtype]

Hey Lasse, this is really terrific info. Thank you so much for your in-depth response! I will need a bit of time to properly digest it and perhaps respond in-depth, myself.

In general if you want to make a custom profile, I wouldn't implement it in a clone/fork of Font Bakery

So, the more general approach you expect individual foundries to take is to just have their own repo for a profile, and then just point to this profile path when they run FontBakery ?

[graphicore]

So, the more general approach you expect individual foundries to take is to just have their own repo for a profile […]

Yes, though, it really depends. We have Adobe Fonts in Font Bakery. It's probably hard to come up with a good set of rules and decision criteria.

[…] , and then just point to this profile path when they run FontBakery ?

Well, if you make a proper package and and a setup.py and install it you can also point to the module name. You can even put your profile on pypi and install it with pip.

[felipesanches]

My suggestion is that new profiles should be always created as a pull request to fontbakery's repo. The only exception to that should be when there's any motivation to keep the checks private. The goal is to gradually migrate checks from vendor-specific profiles to the universal one (whenever possible), which is meant to be more generally useful to the overall user base.

[graphicore]

The only exception to that should be when there's any motivation to keep the checks private.

Or not having to bother with this project directly, e.g. if you don't want to write tests for your checks or don't want to bother with other contributor duties like PR-reviews and other overhead. Another reason could be, that there's just not enough relevance to your profile. It could even be an off-topic profile, e.g. one that checks something else, like PDF documents. I can think of a hundred reasons why I wouldn't contribute directly.

The goal is to gradually migrate checks from vendor-specific profiles to the universal one (whenever possible), which is meant to be more generally useful to the overall user base.

I'd expect, like the googlefonts profile, that most vendor specific checks are no fit for universal (because they are vendor specific). Though, I see that if the checks are in here already, its a lot easier to migrate and move them around, and that's a good reason.

[jenskutilek]

I have a custom FB profile that is installed as a module. Up until now it ran on Fontbakery 0.9, mainly because keeping track of the moved and changed checks with every update is a hassle.

I'm having a hard time moving my profile to FB 0.12. The docs (Writing profiles) now seem completely outdated. Will the docs be updated eventually?

[felipesanches]

Please check this: https://fontbakery.readthedocs.io/en/latest/developer/writing-profiles.html

I noticed that there is an outdated copy of the documentation under a URL that contains a dash in "font-bakery", while the most up-to-date is in URLs without the dash. I am not sure who originally configured that outdated instance on ReadTheDocs. I'd like to take it down to avoid this kind of problem.

I have just updated our README.md now to fix all links that were pointed to the outdated docs. Please let me know if you find any bad URL I may have not seen yet.

[jenskutilek]

@felipesanches Thank you for guiding me in the right direction :) I was indeed on the wrong readthedocs page.