# Metadata Generator

<!-- status autogenerated section -->
| Status        |           |
| ------------- |-----------|
| Stability     | [alpha]: metrics   |
| Issues        | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aopen%20label%3Acmd%2Fmdatagen%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aopen+is%3Aissue+label%3Acmd%2Fmdatagen) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aclosed%20label%3Acmd%2Fmdatagen%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aclosed+is%3Aissue+label%3Acmd%2Fmdatagen) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@dmitryax](https://www.github.com/dmitryax) |

[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
<!-- end autogenerated section -->

Every component's documentation should include a brief description of the component and guidance on how to use it.
There is also some information about the component (or metadata) that should be included to help end-users understand the current state of the component and whether it is right for their use case.
Examples of this metadata about a component are:

- its stability level
- the distributions containing it
- the types of pipelines it supports
- metrics emitted in the case of a scraping receiver, a scraper, or a connector

The metadata generator defines a schema for specifying this information to ensure it is complete and well-formed.
The metadata generator is then able to ingest the metadata, validate it against the schema and produce documentation in a standardized format.
An example of how this generated documentation looks can be found in [documentation.md](https://github.com/open-telemetry/opentelemetry-collector/blob/main/cmd/mdatagen/internal/samplereceiver/documentation.md).

## Using the Metadata Generator

In order for a component to benefit from the metadata generator (`mdatagen`) these requirements need to be met:

1. A yaml file containing the metadata that needs to be included in the component
2. The component should declare a `go:generate mdatagen` directive which tells `mdatagen` what to generate

As an example, here is a minimal `metadata.yaml` for the [OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver):

```yaml
type: otlp
status:
  class: receiver
  stability:
    beta: [logs]
    stable: [metrics, traces]
```

Detailed information about the schema of `metadata.yaml` can be found in [metadata-schema.yaml](./metadata-schema.yaml).

The `go:generate mdatagen` directive is usually defined in a `doc.go` file in the same package as the component, for example:

```go
//go:generate mdatagen metadata.yaml

package main
```

Below are some more examples that can be used for reference:

- The ElasticSearch receiver has an extensive [metadata.yaml](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/elasticsearchreceiver/metadata.yaml)
- The host metrics receiver has internal subcomponents, each with their own `metadata.yaml` and `doc.go`. See [cpuscraper](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver/internal/scraper/cpuscraper) for example.

You can run `cd cmd/mdatagen && $(GOCMD) install .` to install the `mdatagen` tool in `GOBIN` and then run `mdatagen metadata.yaml` to generate documentation for a specific component or you can run `make generate` to generate documentation for all components.

### Component Config Documentation

The metadata generator supports automatic generation of configuration schemas for components.
This generates JSON Schema files that enable IDE autocompletion, validation, and documentation for component configuration.
In the future it will also generate Go config structs and human-readable documentation for configuration options

To define a configuration schema, add a `config` section to your `metadata.yaml`:

```yaml
type: myreceiver
status:
  class: receiver
  stability:
    beta: [metrics, traces]

config:
  type: object
  properties:
    endpoint:
      type: string
      description: The endpoint to listen on
      default: "localhost:4317"
    timeout:
      type: string
      format: duration
      description: Request timeout duration
      default: "30s"
    tls:
      $ref: go.opentelemetry.io/collector/config/configtls.server_config
  required: [endpoint]
```

The `config` section is based on [JSON Schema standard](https://json-schema.org/) (draft 2020-12) and supports:

- **Standard JSON Schema types**: string, number, integer, boolean, object, array, null
- **Validation constraints**: minLength, maxLength, pattern, minimum, maximum, enum, etc.
- **References**: Internal (`$ref: definition_name`), external (`$ref: package.path.type`), or relative (`$ref: ./internal/config.type`)
- **Reusable definitions**: Define common schemas in `$defs` and reference them with `$ref`
- **Schema composition**: Use `allOf` for complex configurations

### Metrics Builder Configuration

For receivers, scrapers, and other components that emit metrics, `mdatagen` can generate metrics builder
configuration from `metadata.yaml`.

```yaml
type: myreceiver
status:
  class: receiver
  stability:
    beta: [metrics]

resource_attributes:
  transport:
    description: Transport used by the request.
    type: string
    enabled: true

attributes:
  status_code:
    description: Response status code.
    type: int
    requirement_level: opt_in

metrics:
  http.server.request.count:
    enabled: true
    description: Number of received requests.
    unit: "{request}"
    sum:
      value_type: int
      monotonic: true
      aggregation_temporality: cumulative
    attributes: [status_code]
```

This lets users:

- enable or disable individual metrics
- enable or disable resource attributes
- use `metrics_include` and `metrics_exclude` on resource attributes to only emit metrics with matching resource attribute values

### Metric Reaggregation Configuration

`mdatagen` lets users reduce metric cardinality by dropping selected metric attributes
and aggregating the resulting datapoints.

```yaml
attributes:
  transport:
    description: Transport used by the request.
    type: string
    requirement_level: recommended
  status_code:
    description: Response status code.
    type: int
    requirement_level: opt_in
```

This adds two per-metric settings for metrics that declare attributes:

- `attributes`: the subset of metric attributes to keep in the emitted metric stream
- `aggregation_strategy`: how collapsed datapoints are merged, using `sum`, `avg`, `min`, or `max`

Defaults:

- sum metrics use `sum`; gauge metrics use `avg`
- `required` attributes are always kept
- `recommended` and `conditionally_required` attributes are kept by default, but users can remove them
- `opt_in` attributes are omitted by default, so that dimension is aggregated unless the user adds it

Example user configuration:

```yaml
receivers:
  myreceiver:
    metrics:
      http.server.request.count:
        enabled: true
        aggregation_strategy: sum
        attributes: [transport]
```

In this example, datapoints that only differ by `status_code` are aggregated together, while
`transport` remains part of the output identity.

### Feature Gates Documentation

The metadata generator supports automatic documentation generation for feature gates used by components. Feature gates are documented by adding a `feature_gates` section to your `metadata.yaml`:

```yaml
type: mycomponent
status:
  class: receiver
  stability:
    beta: [metrics, traces]

feature_gates:
  - id: receiver.mycomponent.newFeature
    description: 'Enables new feature functionality that improves performance'
    stage: alpha
    from_version: 'v0.100.0'
    reference_url: 'https://github.com/open-telemetry/opentelemetry-collector/issues/12345'

  - id: receiver.mycomponent.stableFeature
    description: 'A feature that has reached stability'
    stage: stable
    from_version: 'v0.90.0'
    to_version: 'v0.95.0'
    reference_url: 'https://github.com/open-telemetry/opentelemetry-collector/issues/11111'
```

This will generate a "Feature Gates" section in the component's `documentation.md` file with a table containing:

- **Feature Gate**: The gate identifier
- **Stage**: The lifecycle stage (alpha, beta, stable, deprecated)
- **Description**: Brief description of what the gate controls
- **From Version**: Version when the gate was introduced
- **To Version**: Version when stable/deprecated gates will be removed (if applicable)
- **Reference**: Link to additional contextual information

The feature gate definitions should correspond to actual gates registered in your component code using the [Feature Gates API](../../featuregate/README.md).

By default, mdatagen applies strict validation to feature gate entries:

- Every gate `id` must be prefixed with `<status.class>.<type>.` (e.g. `receiver.mycomponent.newFeature`), so gates are namespaced to the component that owns them.
- Every `reference_url` must be a GitHub issue URL of the form `https://github.com/<owner>/<repo>/issues/<number>`. Pull requests, blog posts, and other URLs are rejected.

Individual gates that predate these rules can be grandfathered in by setting `skip_strict_validation: true` on the gate entry itself. New gates should not set this flag.

### Generate multiple metadata packages

By default, `mdatagen` will generate a package called `metadata` in the `internal` directory. If you want to generate a package with a different name, you can use the `generated_package_name` configuration field to provide an alternate name.

```yaml
type: otlp
generated_package_name: customname
status:
  class: receiver
  stability:
    beta: [logs]
    stable: [metrics, traces]
```

The most common scenario for this would be making major changes to a receiver's metadata without breaking what exists. In this scenario, `mdatagen` could produce separate packages for different metadata specs in the same receiver:

```go
//go:generate mdatagen metadata.yaml
//go:generate mdatagen custom.yaml

package main
```

With two different packages generated, the behaviour for which metadata is used can be easily controlled via featuregate or a similar mechanism.

## Contributing to the Metadata Generator

The code for generating the documentation can be found in [loader.go](./internal/loader.go) and the templates for rendering the documentation can be found in [templates](./internal/templates).
When making updates to the metadata generator or introducing support for new functionality:

1. Ensure the [metadata-schema.yaml](./metadata-schema.yaml) and [metadata.yaml](./metadata.yaml) files reflect the changes.
2. Run `make mdatagen-test`.
3. Make sure all tests are passing including [generated tests](./internal/samplereceiver/internal/metadata/generated_metrics_test.go).
4. Run `make generate`.
