How to create, install, and publish a new Pygments Style

I like pretty colors, and when I’m coding… I like pretty colors in my code. But syntax highlighting is not just about pretty colors, it helps developers to navigate the code and catch errors, by giving a consistent look to the different structures that make up a codebase. In comes Pygments, a syntax highlighter for use in code hosting, forums, wikis or other applications that need to prettify source code. It supports 573 languages and text formats and, at the time of writing, there are 47 Pygments styles. Among them, community favorites such as Atom, Gruvbox and Nord. But what if you want to use colors from other themes? Say… Rose Pine, Ayu or Cattppuccin. Then this guide is for you. I’ll go thought the steps I took to create the Rose Piné Pygments Style (link to repo). How to create a style, assign colors, test, set as plugin and publish.

Creating and installing a Pygments Style

Setting up your environment

Pygments is written in Python, and according to Pygments’s documentation, new styles can be added as plugins through entry points. So in essence, a new style is a small Python package. So we start like any other python project, with the virtual environment. I use Pyenv-virtualenv to create and manage environments and Poetry to manage dependencies, but you can use whatever you feel most comfortable with, although some steps might vary.

The main dependency of the project is Pygments which can be added with:

poetry add pygments

There are other dependencies that we will be using for documentation, listing, checking and testing like: pytest, darglint, coverage, flake8, black, bandit, sphinx and their extension justo name a few, and you add them to your dev group with the command.

poetry add -G dev (name of the package)

This is why I normally use a template to start a project, which takes care of all the development dependencies needed for the backstage. My favorite template these days is Hypermodern Python Cookiecutter by Claudio Jolowicz, which uses the Cookiecutter, command line utility to create the project templates.

Then, at the start of the project the file structure looks like this:

pygments-rose-pine
├── .github/
├── tests/
├── src/
├── docs/
├── .gitignore
├── tox.ini
├── poetry.lock
├── CHANGELOG.md
├── README.md
├── CONTRIBUTING.md
├── CODE_OF_CONDUCT.md
├── LICENSE
├── pyproject.toml
└── codecov.yml

Creating the Style

Now for the style, the Rose Piné theme has three color palettes: Rose Piné, Rose Pine Moon and Rose Piné Dawn, which we’ll be implementing. Inside the src folder we’ll create a file for each of the pallets, like this:

src
├── __init__.py
├── rosepine.py
├── rosepinemoon.py
└── rosepinedawn.py

The name of the files is important since it’s what will helps Pygments find the new style once the plugin is installed. For the styles themselves, according to the Pygments documentation all you have to do is subclass the Style class and define styles. To figure out how to do that I looked, into Reilly Siemmens’s Gruvbox theme for IPython, and in the case of the Rose Piné Moon palette, this is how the file rosepinemoon.py looks like.

from dataclasses import dataclass
from typing import Any

from pygments.style import Style
from pygments.token import (
    Comment,
    Error,
    Keyword,
    Name,
    Number,
    Operator,
    String,
    Text,
    Token,
)


@dataclass(frozen=True)
class Color:
    """Absolute colors as defined by rose-pine: https://rosepinetheme.com/palette."""  # noqa: E501

    moon_base = "#232136"
    moon_surface = "#2a273f"
    moon_overlay = "#393552"
    moon_muted = "#6e6a86"
    moon_subtle = "#908caa"
    moon_text = "#e0def4"
    moon_love = "#eb6f92"
    moon_gold = "#f6c177"
    moon_rose = "#ea9a97"
    moon_pine = "#3e8fb0"
    moon_foam = "#9ccfd8"
    moon_iris = "#c4a7e7"
    moon_highlightlow = "#2a283e"
    moon_highlightmed = "#44415a"
    moon_highlighthigh = "#56526e"


class RosePineMoonStyle(Style):
    """Soho vibes for Pygments. Based on the colors of Rose Pine Moon theme."""

    styles: dict[Any, str] = {
        Comment: Color.moon_subtle,
        Error: Color.moon_love,
        Keyword.Namespace: Color.moon_pine,
        Keyword.Constant: Color.moon_rose,
        Keyword.Type: Color.moon_foam,
        Keyword: Color.moon_pine,
        Name.Builtin.Pseudo: Color.moon_love,
        Name.Builtin: Color.moon_text,
        Name.Class: Color.moon_foam,
        Name.Decorator: f"{Color.moon_iris} bold",
        Name.Exception: Color.moon_foam,
        Name.Function: Color.moon_love,
        Name.Variable.Magic: Color.moon_love,
        Name: Color.moon_text,
        Number: Color.moon_rose,
        Operator.Word: Color.moon_pine,
        Operator: Color.moon_pine,
        String.Affix: Color.moon_pine,
        String.Escape: Color.moon_pine,
        String.Interpol: Color.moon_text,
        String: Color.moon_gold,
        Text: Color.moon_text,
        Token.Punctuation: Color.moon_subtle,
    }

The first section corresponds to the imports for dataclass and typing, then the Pygments Style to be subclassed and the tokens to assign colors to keywords, numbers, comments, etc. Below that, two classes are defined: One for the colors, and another for the Style. To know which color goes where, I used the specific colors from palette as defined in the Rose Piné Theme page, and Pygment’s Token Guide.

Installing your Pygments style

Now that you have your custom style, you need to install it by registering it as a plugin. This can be done from the pyproject.toml file or setup.py file if your project uses that file. This project uses the pyproject.toml file, so to register the plugin I used the [tool.poetry.plugins."pygments.styles"] table. Also, one thing to note is that, on the [tool.poetry] table, I registered the style files as packages this way I can reference the files directly on the plugins table.

[tool.poetry]
…
packages = [
    { include = "rosepine.py", from = "src" },
    { include = "rosepinemoon.py", from = "src" },
    { include = "rosepinedawn.py", from = "src" },
]

…

[tool.poetry.plugins."pygments.styles"]
"rose-pine" = "rosepine:RosePineStyle"
"rose-pine-moon" = "rosepinemoon:RosePineMoonStyle"
"rose-pine-dawn" = "rosepinedawn:RosePineDawnStyle"

After that, you can run the poetry install command to install the local package.

Testing and Troubleshooting

“If it’s not tested, it’s broken.”

Bruce Eckel

Now we are ready to test, but… how do we test colors? The manual approach is to use the newly installed theme and see what it looks like on a sample of code. I did that with IPython, and this was the result.

But for a more programatic approach, testing can be done by comparing the output from the function against an ANSI string with the desired result. To do that, inside the tests files, we can create the test_rosepine.py file to test our new styles. We’ll be using Pytest for testing , and some functions and classes from the Pygments package, so we’ll import those first, as well as the styles we have created.

import pygments
import pytest

from pygments.lexers import PythonLexer
from pygments.formatters import TerminalTrueColorFormatter
from pytest import param

from rosepine import RosePineStyle
from rosepinemoon import RosePineMoonStyle
from rosepinedawn import RosePineDawnStyle

Then, for each style, we’ll create a function that highlights a string of code. The example below is for the Rose Piné Moon theme.

def highlight_moon(code: str) -> str:
    """Highlight code in true color (24-bit) with the RosePineMoonStyle."""
    return pygments.highlight(
        code=code,
        lexer=PythonLexer(),
        formatter=TerminalTrueColorFormatter(style=RosePineMoonStyle),
    )

Now we can create our tests. The concept is simple, but there are a ton of tokens to test, so I’ll be using the @pytest.mark.parametrize decorator to tests agains all cases. Below it’s an example, for comments.

@pytest.mark.parametrize(
    "code,expected",
    [
        param(
            "# This is a comment.",
            "\x1b[38;2;144;140;170m# This is a comment.\x1b[39m\n",
            id="Comment",
        ),
        …
    ],
)
def test_highlighting_moon(code: str, expected: str) -> None:
    """The given code should highlight as expected."""
    assert highlight_moon(code) == expected

The decorator takes a string with the variables to parametrize separated by commas as the first argument, and a list of the parameters as the second. The use of the param function was also informed by Reilly Siemmens’s Gruvbox theme‘s testing suit, and while testing, I found out just how useful is the id argument. It can help you know which of the parameters failed while testing, which is specially important in this case, since there were a lot of them. To run the tests you can use.

pytest -cov

This will run pytest, and coverage, to see what amount of code is covered by your tests.

Publishing your Pygments style

Preparing your style for publication

Once the testing is done we are ready right? No. Before we can start sharing our creation with the world we need to make sure it’s understandable and well documented. That’s where CONTRIBUTING.md, CHANGELOG.md, CODE OF CONDUCT.md, LICENSE.md and README.md files come in. This is a small plugin, so the installation, usage and requirements can be specified in the README, and it might be enough. But if you are working on a larger project, keeping a documentation folder hosted on a platform like Read the Docs or GitHub Pages is something that needs to be done. Below are some steps if you are interested in getting some practice setting up documentation.

sphinx-quickstart

.. |rose-pine-logo| image:: https://github.com/rose-pine/rose-pine-theme/raw/main/assets/icon.png    :width: 30

=======================================
|rose-pine-logo| Rose Piné for Pygments
=======================================

All natural pine, faux fur and a bit of soho vibes for the classy minimalist

|Python versions| |Latest release| |test| |Codecov| |License| |Black| |Rose Pine Community|

.. toctree::    
   :maxdepth: 1
   :caption: Contributing
   :hidden:

   changelog.rst
   contributing.rst
   code-of-conduct.rst
   license.rst
   
 Installation
 ============
 
 ...

docs
├── static
├── _build
├── conf.py
├── index.rst
├── contributing.rst
├── code-of-conduct.rst
├── license.rst
└── changelog.rst

Another thing to keep in mind is making sure our code is well formatted and follows PEP guidelines, for that we can use the flake8 and black packages.

Once all that is done, we are ready to build our first release. Thankfully Poetry makes that easy with the build command.

poetry build

This will create a dist folder with two files inside.

dist
├── pygments_rose_pine-1.0.0.tar.gz
└── pygments_rose_pine-1.0.0-py3-none-any.whl

These are the files that can be installed when someone wants to use the style.

Sharing your new Pygments style

The first way of sharing your style is through GitHub, you can create a release and give people instructions in your README. The second it through PyPy, the Python Packaging Index, so people can install it using pip.

To create a release, you can go to your repo and under releases click on Create a new release. There you can tag it using Semantic Versioning to identify your release, and drop the files you you created on the dist folder.

Also, if your theme is part of a larger theme, like Rose Piné, you could write to the owner or organization to let them know you have created a new theme.

And that’s it, you can now create your own Pygments styles in the colors you like best. Also, this is a very good example of how to start and practice all the steps that go into creating and publishing open source packages. You can go crazy with this really, for example, you can do Continuous Integration with GitHub Actions or Travis CI, or use Codecov to measure coverage. All the bells and whistles.

Over and out… Happy coding!

Andrea