The Marker Book

Marker is an experimental code analysis interface with the goal to create a stable and user-friendly linting framework for the Rust programming language. This book documents everything related to Marker, from use instructions over lint development to the inner workings of Marker.

Note

The project and documentation is in the early stages of development. If you find that any documentation is outdated, please create an issue so we can fix it.

Usage

This section is intended for users who want to install Marker and run provided lints.

Installation

Marker is split into several components. This section covers the installation of the cargo marker sub-command and the installation of a driver, which is the backend needed to parse the source code.

• Prerequisite
  • Download pre-compiled binaries (recommended)
• Build cargo marker plugin from sources
• Driver and Toolchain
  • Automatic Installation
• Manual Installation
  • Toolchain
  • Driver Compilation
  • Driver Location
• Driver Selection

Prerequisite

The marker sub-command is provided by the cargo_marker crate. This crate requires Cargo and rustup to be installed. Currently, only Unix and Windows systems are supported. Ubuntu, Windows, and macOS are actively tested in the CI.

Download pre-compiled binaries (recommended)

We provide pre-compiled binaries for the mainstream platforms. See the list of available artifacts in our Github Releases.

Select one of the installation scripts below according to your platform. The script will install the required Rust toolchain dependency on your machine, download the current version of cargo-marker CLI, and the internal driver.

Linux or MacOS (Bash):

curl -fsSL https://raw.githubusercontent.com/rust-marker/marker/v0.5/scripts/release/install.sh | bash

Windows (PowerShell):

curl.exe -fsSL https://raw.githubusercontent.com/rust-marker/marker/v0.5/scripts/release/install.ps1 | powershell -command -

The provided scripts use a sliding git tag v0.5, to allow for automatic patch version updates, however a fixed tag v0.5.0 is also available.

If you are on a platform that isn't supported yet by the pre-compiled binaries, then you should fall back to building from sources as described below.

Build cargo marker plugin from sources

Marker provides a new Cargo sub-command, that handles the driver installation, lint crate compilation, and checking process for you. To install it, simply use:

cargo install cargo_marker

You now have the new cargo marker command installed.

Driver and Toolchain

Marker requires a driver, which handles the lexing, parsing, and type checking to then translates everything into Marker's API representation to be checked by the lint crates.

Automatic Installation

Marker's rustc driver requires a specific nightly toolchain to be installed. The nightly version is updated every six weeks when a new version of Rust and Marker is released. You, as the user, should not notice this requirement once it's installed, as cargo marker handles everything for you. To automatically install the driver and toolchain, you can simply run the following command:

cargo marker setup --auto-install-toolchain

Manual Installation

It's highly recommended to use the automatic installation method described above. This is a guide for cases where this is not possible for one reason or another. Note that the manual installation is not actively tested. If you encounter any issues, you're welcome to report them.

Toolchain

The rustc driver requires a specific nightly version to be built and started. The specific nightly version can be found in the README.md of the driver you want to install. The compilation requires at least the following components: rustc, rustc-dev, llvm-tools. These can be installed for the specific toolchain with the following rustup command:

# Fill in the $toolchain variable
rustup toolchain install $toolchain --component rustc-dev llvm-tools

Driver Compilation

The build script of the driver has a check to prevent accidental compilation of the driver. To manually compile the driver, the MARKER_ALLOW_DRIVER_BUILD environment value has to be set. This simply shows the driver that this is not accidental and disables the check. Then it can be compiled as usual. Here is the cargo command:

# Make sure to run cargo with the correct toolchain.
MARKER_ALLOW_DRIVER_BUILD=1 cargo build --release

Driver Location

By default, the driver is stored with the toolchain that it was built with. This makes it simple to find the right driver for a given toolchain and allows the installation of multiple drivers for different toolchains. You can find the toolchain folder using the following rustup command:

# Fill in the $toolchain variable
rustup +$toolchain which cargo

The driver should be located next to the cargo binary, whose path was given by the previous command. If you're not using rustup, you can store the driver binary next to the cargo-marker file. If you're invoking the driver directly, you have to make sure to provide the required libraries to run the driver.

Driver Selection

The cargo marker command searches several locations for the driver and selects the first one it finds. The following locations are searched:

1. The toolchain that was used for the cargo marker command.
2. The toolchain that is hard coded in the cargo-marker binary. (Updated every six weeks with a new release of the driver and cargo_marker crate)
3. Any driver stored next to the cargo-marker binary file.

CI

Marker's primary objective is to offer an excellent linting interface, including the seamless integration with CI services. This document outlines the available CI tooling and provides example templates.

• GitHub Action
  • Git tags
  • Inputs
  • Environment variables
  • Example workflows
    • Basic usage
    • Advanced usage
  • GitHub-managed runners
  • Self-hosted runners
• Other CI systems

GitHub Action

Marker provides a GitHub Action that downloads the pre-compiled binaries and runs cargo marker.

- uses: rust-marker/marker@v0.5

Git tags

The git tag specified in the GitHub Action indicates which version of Marker should be installed. There are several tag flavors available:

• Sliding tags, like v0.5 (recommended):

  Use this to get automatic patch updates.

• Fixed tags, like v0.5.0:

  Use this to pin a specific patch version. If you find a regression in a patch version, please create a new issue. Patch versions must never break anything!

⚠️ The minor versions before Marker v1 contain breaking changes. While there is a sliding v0 tag, it's highly recommended to include the minor version as well. This prevents uncontrolled CI breakage with every release.

Inputs

All inputs are optional, they only allow tweaking the default behavior.

Environment variables

These environment variables configure the behavior of the installation script and they may be used if you run that script directly as well e.g. on other CI systems.

Example workflows

These example workflows will use the lint crates specified in the Cargo.toml file by default. Refer to the Lint Crate Declaration section for more information.

Basic usage

Checkout the repository code, install the toolchain, Marker, and start linting.

jobs:
  rust-marker-lints:
    runs-on: ubuntu-latest
    env:
      RUSTFLAGS: --deny warnings
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rust-lang/setup-rust-toolchain@v1
      - uses: rust-marker/marker@v0.5

Advanced usage

If you need something more than just the cargo marker command, you may use the action to only install Marker and then manually run the cargo marker command, just like in your local dev environment.

Here is an example of how you could limit the set of crates that you want to lint. Refer to cargo marker --help for a full list of available options.

jobs:
  rust-marker-lints:
    runs-on: ubuntu-latest
    env:
      RUSTFLAGS: --deny warnings
    steps:
      - uses: actions/checkout@v4
      - uses: actions-rust-lang/setup-rust-toolchain@v1
      - uses: rust-marker/marker@v0.5
        with:
          install-only: true
      - run: cargo marker -- -p crate-foo -p crate-bar

If you have an example of advanced usage of cargo marker command that you have to repeat in your CI template again and again consider opening a new issue in our repository. We will be glad to hear any suggestions about extending the inputs for the GitHub Action for your use case.

GitHub-managed runners

The action is cross-platform. It supports Windows, Linux and MacOS. It is tested on all OS images supported by managed GitHub Actions runners.

If GitHub releases a newer OS image version it is very likely that this action will still "just work" on it. We make sure to add new OS images that GitHub-managed runners support to our CI test suite, as well as remove support for the old ones following the GitHub Actions's OS images deprecation cadance.

Self-hosted runners

The action must work on self-hosted runners out of the box as well. We don't test all possible operating systems on our CI, but if your self-hosted runners use the OS version from the range of OS images supported by managed GitHub Actions runners or other compatible distributions, then it should work. If you have some exotic OS setup which breaks our GitHub Action we will be interested to hear about that in a new issue.

Other CI systems

If you don't use GitHub Actions CI, you can still benefit from the pre-made installation scripts that automate the downloading of the pre-compiled binaries on CI for you.

These curl commands differ slightly from the scripts mentioned in the installation chapter. They are more verbose for additional readability in the CI templates, and they also contain additional options to retry spurious network errors for stability on CI.

You can run these scripts on any CI system of your choice, and they will make the cargo marker command available for you.

Linux or MacOS (Bash):

curl \
    --location \
    --silent \
    --fail \
    --show-error \
    --retry 5 \
    --retry-all-errors \
    https://raw.githubusercontent.com/rust-marker/marker/v0.5/scripts/release/install.sh \
    | bash

Windows (PowerShell):

curl.exe `
    --location `
    --silent `
    --fail `
    --show-error `
    --retry 5 `
    --retry-all-errors `
    https://raw.githubusercontent.com/rust-marker/marker/v0.5/scripts/release/install.ps1 `
    | powershell -command -

Both of these scripts are configurable. See the environment variables for details on what's available.

The available version git tags that you may use in the URL are described in the git tags paragraph of the Github Action.

Lint Crate Declaration

Marker in itself, is a linting interface. The actual code analysis is implemented in external crates compiled to dynamic libraries, called lint crates. This section covers how these lint crates can be specified. Once they're specified, they will be automatically fetched and build by the cargo marker command, to then be loaded by the driver for linting.

• Declaration in Cargo.toml
• Declaration as arguments

Declaration in Cargo.toml

The main way to declare lint crates, is to add them to the Cargo.toml file under the [workspace.metadata.marker.lints] section. There they can be defined like a normal dependency, with a version, git repository, or path. This is a short example of the three methods:

[workspace.metadata.marker.lints]
# An external crate from a registry
marker_lints = "0.5.0"

# An external crate from git
marker_lints = { git = "https://github.com/rust-marker/marker" }

# A local crate as a path
marker_lints = { path = './marker_lints' }

Declaration as arguments

Lints can also be declared as arguments to the cargo marker command. Marker will skip reading the Cargo.toml file if any lint crate was specified this way. This is intentional, to allow tools to use Marker for lexing and parsing, regardless of the normally specified lint crates.

A lint crate can be specified with the --lints option. The string is expected to have the same format, that would be used in the Cargo.toml file. Here is an example for the same lint crates specified above:

# An external crate from a registry
cargo marker --lints "marker_lints = '0.5.0'"

# An external crate from git
cargo marker --lints "marker_lints = { git = 'https://github.com/rust-marker/marker' }"

# A local crate as a path
cargo marker --lints "marker_lints = { path = './marker_lints' }"

Setting Lint Levels

Once your crate is configured to run some lints, it quickly becomes important to control the lint levels within your code.

Marker provides the ability to use normal lint control attributes #[allow(...)], #[deny(...)], and #[warn(....)] to change the behavior of marker lints. One caveat is that you need to put these attributes under a conditionally-compiled attribute #[cfg_attr(marker, ...)].

Example:

#![allow(unused)]
fn main() {
#[cfg_attr(marker, allow(marker::lint_crate::lint_name))]
fn foo() {}
}

Lints namespacing

Marker uses the marker:: tool prefix for lints. This is to make sure that your lints never collide with the native rustc lints and lints from any other linting tools. This is similar to how clippy puts all of its lints under clippy:: prefix.

After marker:: there must be the name of the lint crate. This is to make sure that lints from different lint crates never collide with each other. The name of the lint crate must be in snake case. The rule is the same as when you reference the crate in your code as a dependency. If the name of the crate contains dashes, they will be replaced with underscores.

The last segment is the name of the lint itself, which is the lowercaed name of the static variable that defines it in the lint crate.

Conditional compilation

There is a problem that a regular cargo check/build knows nothing about Marker and it will complain about unknown lints unless marker-specific attributes are compiled-out. To work around this Marker passes a --cfg=marker flag that you can use in your code.

This allows you to conditionally include allow/warn/deny attributes such that only cargo marker sees them, and regular builds ignore them allowing you to continue using the version of the Rust toolchain you are using in your project.

Per-lint-crate cfg

Marker also passes --cfg=marker="lint_crate" for each loaded lint crate that participates in linting. You may use this flag to conditionally control the level of lints from a specific lint crate.

This is intended to be used only for lints that are time consuming. For example, you could group lints that take a lot of time to run in a separate lint crate. This way you can run Marker with these additional lints enabled on CI, but without them locally.

Example:

Run lints from a heavy_lint_crate only on CI.

cargo marker --lints 'heavy_lint_crate = 1.0.0'

Use the default lints from Cargo.toml metadata locally.

cargo marker

Conditionally ignore the lint from the heavy_lint_crate, but only if it actually participates in linting.

#![allow(unused)]
fn main() {
#[cfg_attr(marker = "heavy_lint_crate", allow(marker::heavy_lint_crate::lint_name))]
fn foo() {}
}

This way you'll avoid triggering an "unknown lint" error if the heavy_lint_crate isn't included during the linting.

The reason for doing this kind of conditional compilation instead of just including heavy_lint_crate in linting unconditionally is a bit intricate. If you put any allow attributes in your code for the heavy lints then this won't prevent them from running. Of course, the lints for code under an allow will not be emitted, but the lints logic still walks through that code. Therefore, if you want to prevent a heavy lint from running you should just not include it in the lints configured in Cargo.toml or with --lints CLI parameter.

Lint Crate Security

Marker's lint crates are normal crates compiled into dynamic libraries. The compilation is done with Cargo, which will execute the build script and proc macros used by the lint crates. This means that lint crates can theoretically perform malicious activities during the compilation and checking processes. It's advised to only run lint crates from trusted sources, perform audits of used crates, or use a sandbox.

Malicious behavior in lint crates

We're sadly unable to review and validate all lint crates. Marker also doesn't host any lint crates itself. Instead, it relies on existing infrastructure like git repositories and crate registries. If you notice malicious behavior in a lint crate, please handle it like you would with any other dependency. It's recommended to take the following steps:

1. Avoid using the lint crate until the issue has been resolved.
2. If the malicious behavior seems to be incidental, contact the author to fix the issue.
3. If the malicious behavior seems to be intentional, report the lint crate according to the policy of the hosting platform.

Sandboxing

Lint crates are loaded as normal dynamic libraries. Marker sadly doesn't provide any option to sandbox the lint crates. If you want to use an untrusted lint crate, it's recommended that you manually sandbox the entire Marker process.

Lint Development

This section will cover everything you need to know to develop a bullet proof lint crate.

If you just want a quick start, you can checkout Marker's lint-crate template

Marker Development

This section will document the architecture and inner workings of Marker.

If you're interested in contributing to Marker, please also check out:

• Marker's Code of Conduct
• Marker's Contribution documentation