provisioning

134 notes in this subject, read out of this brain and free to use. This is page 2 of 3.

Git Sync max_repositories configuration setting

The max_repositories configuration setting controls how many repositories you can sync on self-managed Grafana. The default is 10, but you can set it to 0 for unlimited repositories. This setting is located in the [provisioning] section of the Grafana configuration file.

Git Sync max_resources_per_repository configuration setting

The max_resources_per_repository configuration setting controls the amount of resources per repository to sync on self-managed Grafana. The default is 0, which means unlimited. This is a performance recommendation, not a hard cap. You can customize this limit through configuration settings on-prem.

Scale beyond 1000 resources per repository

If a single repository holds more than 1,000 resources, you can connect the same repository multiple times, with each connection pointing to a different folder (path) in the repository. Each connection syncs its own subset of resources and counts toward the 1,000-resource recommendation independently. A stack allows up to 10 repository connections, which lets you sync up to roughly 10,000 resources from a single repository while keeping each connection within the recommended range. Use folderless sync for each connection to avoid creating nested wrapper folders named after the repository.

Git Sync across multiple organizations

Git Sync works across multiple organizations for self-managed Grafana instances. You can set up Git Sync independently in each of your organizations, and what you sync in one organization does not affect another. This feature is available starting in Grafana 13.0.4, but not supported until Grafana 13.1.1.

Git Sync compatible providers table

Git Sync is available for multiple Git providers with varying authentication methods. Pure Git: available in Cloud, OSS, and Enterprise; uses Personal Access Token authentication over HTTPS Smart HTTP protocol v2. GitHub: available in Cloud, OSS, and Enterprise; supports Personal Access Token or GitHub App authentication. GitLab: available in Cloud and Enterprise; uses Personal Access Token authentication. Bitbucket: available in Cloud and Enterprise; uses API token with scopes authentication. Pure Git, GitLab, and Bitbucket are supported in Grafana v12.4.x or later only.

Pure Git protocol requirements

Pure Git only supports Smart HTTP protocol v2 (Git over HTTPS). Earlier protocol versions (v1, v0) and SSH transport are not supported. You must make sure your Git server supports protocol v2 over HTTPS. Some providers like Azure DevOps only use v1 and are not compatible with Git Sync. Pure Git doesn't include provider-specific features that require provider APIs, such as webhook-driven instant sync, automated PR comments, or deep links to source files.

Enhanced Git integrations GitHub GitLab Bitbucket

Enhanced integrations for GitHub, GitHub Enterprise, GitLab, and Bitbucket allow workflows that feel native to those platforms: automated pull request comments with dashboard previews, instant webhook-based sync, or direct navigation from Grafana to source files in the provider's UI. The GitHub enhanced integration is the most feature-complete experience, enabling richer pull request workflows and tighter integration into review processes. It is available in Grafana OSS, Enterprise, and Cloud. GitLab and Bitbucket integrations have limited functionality and are only available in Grafana Enterprise and Grafana Cloud, with continued improvements expected in upcoming releases.

Git Sync supported resources

Git Sync only supports dashboards and folders. Alerts, data sources, panels, and other resources are not supported yet.

Git Sync resource compatibility status

A resource can be compatible or incompatible. Supported resources that are compatible can be managed with Git Sync. Supported resources that are incompatible cannot be managed with Git Sync due to data format issues; syncing will be blocked until compatibility is fixed with a migration tool. Unsupported resources cannot be managed with Git Sync regardless of compatibility status.

Git Sync full-instance sync is experimental

Full-instance sync is experimental in Git Sync. When migrating resources to Git Sync, you can still create, edit, or delete resources, but changes may not be exported. The duration of this process depends on the number of resources involved. When migrating existing dashboards, the folder structure will be replicated in the repository, and you may need to manually remove or manage original folders after the migration.

Shard by capacity not by team for scaling Git Sync

When you have many teams or tenants, do not create one connection per team. Instead, shard by capacity: create one repository and group teams into a small number of shard folders, each holding up to about 1,000 resources, and connect each shard folder separately. This approach consumes connections efficiently, scales as teams grow, and on Grafana Cloud avoids requiring the hundreds of connections that one-per-team would need. You can move teams between shards at any time to balance the load, and only pay for the connections you actually need.

Modify Git Sync usage limits on Grafana Cloud

On Grafana Cloud, limit increases are not granted automatically. When you request a limit increase, Grafana Support will ask you to describe your use case to understand why current limits are insufficient and assess whether the increase is necessary and safe for your stack's performance. The 10-connection limit can be increased slightly on request by contacting Support. Splitting a single repository across multiple connections is the recommended way to sync more resources without changing tier limits.

Modify Git Sync usage limits on-prem

On self-managed Grafana (OSS or Enterprise), you can change usage limits directly through configuration settings without making a request. The table value of 10 repositories is only the default, not a hard ceiling. Resources per repository remain unlimited by default. The 1,000-resources-per-connection figure is a performance recommendation, not an on-prem default cap. You can customize both limits through configuration settings. Prefer splitting a single repository across connections when you need more resources, rather than raising max_resources_per_repository far above the recommended range.

Graphite data source YAML provisioning example

Example Graphite YAML provisioning file: apiVersion: 1 datasources: - name: Graphite type: graphite access: proxy url: http://localhost:8080 jsonData: graphiteVersion: '1.1'

Prometheus Terraform provisioning example - with authentication

Example Terraform configuration for Prometheus data source with basic authentication: ```hcl resource "grafana_data_source" "prometheus_auth" { name = "Prometheus (authenticated)" type = "prometheus" url = "https://prometheus.example.com:9090" basic_auth_enabled = true basic_auth_username = "grafana" json_data_encoded = jsonencode({ httpMethod = "POST" prometheusType = "Mimir" timeInterval = "15s" }) secure_json_data_encoded = jsonencode({ basicAuthPassword = var.prometheus_password }) } ```

Prometheus Terraform provisioning example - basic

Example Terraform configuration for Prometheus data source with common settings: ```hcl resource "grafana_data_source" "prometheus" { name = "Prometheus" type = "prometheus" url = "http://localhost:9090" json_data_encoded = jsonencode({ httpMethod = "POST" manageAlerts = true prometheusType = "Prometheus" prometheusVersion = "3.3.0" cacheLevel = "High" disableRecordingRules = false incrementalQuerying = true incrementalQueryOverlapWindow = "10m" timeInterval = "15s" exemplarTraceIdDestinations = [{ datasourceUid = "my_tempo_uid" name = "traceID" }] }) } ```

Prometheus YAML provisioning example

Example Prometheus data source YAML configuration: ```yaml apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://localhost:9090 jsonData: httpMethod: POST manageAlerts: true allowAsRecordingRulesTarget: true prometheusType: Prometheus prometheusVersion: 3.3.0 cacheLevel: 'High' disableRecordingRules: false seriesEndpoint: false timeInterval: 10s incrementalQueryOverlapWindow: 10m exemplarTraceIdDestinations: - datasourceUid: my_jaeger_uid name: traceID - name: traceID url: 'http://localhost:3000/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Jaeger%22,%7B%22query%22:%22$${__value.raw}%22%7D%5D' ```

Trace to logs provisioning using tracesToLogsV2 block

You can provision the trace to logs configuration using the tracesToLogsV2 block in your data source YAML file. Refer to provisioning data sources documentation for general provisioning instructions and the Tempo data source provisioning documentation for the full provisioning YAML example including all Tempo settings.

Provisioned data sources cannot be modified in Grafana UI

You cannot modify a provisioned data source from the Grafana UI. If using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to logs, clone the data source to create an editable copy, or update the provisioning file for self-managed instances.

Zipkin Terraform configuration with trace to logs

The following Terraform example includes trace to logs configuration linking traces to a Loki data source: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" json_data_encoded = jsonencode({ tracesToLogsV2 = { datasourceUid = grafana_data_source.loki.uid spanStartTimeShift = "1h" spanEndTimeShift = "-1h" filterByTraceID = true filterBySpanID = false tags = [ { key = "service.name", value = "service" }, { key = "namespace" } ] } nodeGraph = { enabled = true } }) } ```

Zipkin Terraform configuration with basic authentication

The following Terraform example includes basic authentication: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" basic_auth_enabled = true basic_auth_username = "zipkin_user" secure_json_data_encoded = jsonencode({ basicAuthPassword = var.zipkin_password }) json_data_encoded = jsonencode({ nodeGraph = { enabled = true } }) } ```

Zipkin YAML provisioning example

The following YAML example provisions a Zipkin data source with trace to logs and trace to metrics configuration: ```yaml apiVersion: 1 datasources: - name: Zipkin type: zipkin uid: zipkin-ds url: http://localhost:9411 access: proxy basicAuth: true basicAuthUser: my_user readOnly: false isDefault: false jsonData: tracesToLogsV2: datasourceUid: 'loki' spanStartTimeShift: '1h' spanEndTimeShift: '-1h' tags: ['job', 'instance', 'pod', 'namespace'] filterByTraceID: false filterBySpanID: false customQuery: true query: 'method="$${__span.tags.method}"' tracesToMetrics: datasourceUid: 'prom' spanStartTimeShift: '1h' spanEndTimeShift: '-1h' tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }] queries: - name: 'Sample query' query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))' nodeGraph: enabled: true spanBar: type: 'None' secureJsonData: basicAuthPassword: my_password ```

Zipkin basic Terraform configuration

The following Terraform example creates a basic Zipkin data source: ```hcl resource "grafana_data_source" "zipkin" { name = "Zipkin" type = "zipkin" url = "http://localhost:9411" } ```

POST /api/admin/provisioning/*/reload reloads provisioning configurations

The Admin API provides provisioning reload endpoints: POST /api/admin/provisioning/dashboards/reload, POST /api/admin/provisioning/datasources/reload, POST /api/admin/provisioning/plugins/reload, POST /api/admin/provisioning/access-control/reload, and POST /api/admin/provisioning/alerting/reload. These endpoints reload the provisioning config files for the specified type and re-provision entities. The request won't return until new provisioned entities are stored in the database. For dashboards, it stops polling for changes and restarts with new configurations after returning. All return HTTP 200 with message like 'Dashboards config reloaded'.

SCIM UI configuration advantages

The Grafana SCIM UI provides advantages over configuration file setup: it is accessible by Grafana Cloud users, does not require Grafana restart after configuration updates, and uses authentication settings permission to restrict access scope rather than requiring overly permissive Admin role. Configuration changes made through the UI take precedence over settings in the configuration file or environment variables.

SCIM configuration file example

Example SCIM configuration in ini format: [auth.scim] user_sync_enabled = true group_sync_enabled = false reject_non_provisioned_users = false

SCIM provisioning availability

SCIM provisioning is available in Grafana Enterprise and Grafana Cloud only.

SCIM authentication requirement

SCIM provisioning only works with SAML authentication. Other authentication methods are not supported.

SCIM benefits for user and team management

SCIM offers automated user provisioning (create, update, and disable users when changes occur in the identity provider), automated team lifecycle management (create teams when new groups are added, update team memberships, delete teams when groups are removed), reduced administrative overhead, and enhanced security through automatic access disablement when users leave the organization.

SCIM requires userUID SAML assertion

When using SAML with SCIM, the login authentication flow requires the SAML assertion exchange to include the userUID SAML assertion containing the user's unique identifier at the Identity Provider.

Align SAML identifier with SCIM externalId

When using SAML with SCIM provisioning, the SCIM externalId must align with the SAML user identifier. Use a stable IdP attribute (such as Entra ID user.objectid) as the SCIM externalId and send that same value as a SAML claim. Configure Grafana to read this claim with the assertion_attribute_external_uid setting so SAML authentication links to the SCIM-provisioned user and its permissions. If the SAML identifier and SCIM externalId differ, Grafana may not link the authenticated user to the intended SCIM profile, resulting in incorrect access.

Configure SCIM via Grafana UI location

SCIM can be configured in Grafana by navigating to Administration > Authentication > SCIM.

SCIM UI configuration settings

The SCIM UI settings are: Enable User Sync (required, default false) to enable SCIM user provisioning and automatic user creation, update, and deactivation; Enable Group Sync (optional, default false) to enable SCIM group provisioning for automatic team creation, update, and deletion, cannot be enabled if Team Sync is enabled; Reject Non-Provisioned Users (optional, default false) to prevent non-SCIM provisioned users from signing in (Cloud Portal users can always sign in regardless).

SCIM configuration file settings

SCIM configuration file options are: user_sync_enabled (required, default false) to enable SCIM user provisioning for automatic user creation, update, and deactivation; group_sync_enabled (optional, default false) to enable SCIM group provisioning for automatic team creation, update, and deletion, cannot be enabled if Team Sync is enabled; reject_non_provisioned_users (optional, default false) to prevent non-SCIM provisioned users from signing in (Cloud Portal users can always sign in regardless).

SCIM Terraform configuration

SCIM can be configured using the Grafana Terraform provider grafana_scim_config resource with the same options as manual configuration: user_sync_enabled (required, default false), group_sync_enabled (optional, default false), and reject_non_provisioned_users (optional, default false).

SCIM Terraform configuration example

Example Grafana SCIM configuration with Terraform: resource "grafana_scim_config" "scim_config" { user_sync_enabled = true group_sync_enabled = false reject_non_provisioned_users = false }

SCIM supported identity providers

SCIM provisioning is supported with Entra ID and Okta as identity providers.

SCIM and Team Sync compatibility

SCIM group sync (group_sync_enabled = true) and Team Sync cannot be enabled simultaneously. However, SCIM user sync (user_sync_enabled = true) can be used alongside Team Sync.

SCIM synchronization process

The SCIM synchronization process works as follows: configure SCIM in both the identity provider and Grafana; the identity provider sends SCIM requests to the Grafana SCIM API endpoint; Grafana processes these requests to create, update, or deactivate users and teams, and synchronize team memberships.

SCIM compared to other sync methods

SCIM supports full automation for users and teams with role synchronization handled separately through Role Sync, enabling complete user and team lifecycle management with automatic team creation. It requires SAML authentication and is available on both on-premises and Cloud. Team Sync partially syncs team memberships to existing teams but requires manual team creation. Active LDAP Sync provides full automation for LDAP users but is limited to LDAP environments and on-premises only. Role Sync provides full automation of basic role assignment but is limited to basic roles only. Org Mapping provides full automation of basic role assignment per organization but is limited to basic roles and on-premises only.

Security section encryption configuration example

Example [security] section configuration: ``` [security] secret_key = AaaaAaaa encryption_provider = googlekms.example-encryption-key available_encryption_providers = googlekms.example-encryption-key ```

Secrets migration not needed for new Grafana instances

If updating the encryption key during initial setup of Grafana before any data sources or dashboards have been created, the secrets migration step is not necessary because there are no secrets in Grafana to migrate.

Google Cloud KMS prerequisites

To encrypt secrets with Google Cloud KMS, prerequisites include a Google Cloud account with permission to list and create KMS keys and service accounts, and access to the Grafana configuration file.

Google Cloud KMS setup steps

Setup steps for Google Cloud KMS encryption: create a key ring in Google Cloud KMS, create a symmetric encryption key in the key ring, create a service account with a role having permissions to encrypt and decrypt with Key Management Service, create a service account key and save its JSON file, enable envelope encryption within Grafana, add Google Cloud KMS details to grafana.ini, and restart Grafana.

Google Cloud KMS provider configuration fields

A Google Cloud KMS provider section in the grafana.ini file requires two fields: key_id (the encryption key ID from Google Cloud KMS) and credentials_file (the full path to the service account key JSON file).

Enable Google Cloud KMS encryption example

Example of a Google Cloud KMS provider section in grafana.ini: ``` [security.encryption.googlekms.example-encryption-key] key_id = 1234abcd-12ab-34cd-56ef-1234567890ab credentials_file = ~/.config/gcloud/sample-project-credentials.json ```

Security section encryption provider configuration

The [security] section of grafana.ini must be updated with: secret_key (previous encryption key for legacy alerts and decryption of existing secrets), encryption_provider (in format <PROVIDER>.<KEY-NAME>), and available_encryption_providers (space-separated list of configured key providers).

Re-encrypt secrets with Google Cloud KMS

After configuring Google Cloud KMS encryption and restarting Grafana, optionally re-encrypt all existing secrets in the database with the command: grafana cli admin secrets-migration re-encrypt. This process could take several minutes depending on the number of secrets and users may experience errors while it runs.

Google Cloud KMS encryption configuration section format

Add a new section to the grafana.ini configuration file in the format [security.encryption.googlekms.<KEY-NAME>], where <KEY-NAME> is any name that uniquely identifies this key among other provider keys.

Logger storage for usage insights

When using logger as the storage type for usage insights export, logs are printed to the Grafana server log. There are no additional configuration options for the logger storage type.

Usage insights export configuration section

To enable usage insights export, configure the [usage_insights.export] section in the Grafana configuration file with enabled = true and storage = loki or storage = logger.

Usage insights export storage types

Storage type options for usage insights export are loki and logger. The logger option was added in Grafana Enterprise 8.2.

Loki storage configuration for usage insights

When storage type is set to loki, configure [usage_insights.export.storage.loki] with: type (grpc or http, required) - communication protocol; url (required, format host:port) - Loki ingestion server address; tls (boolean, defaults to true) - whether to use secure connection; tenant_id (optional, disabled by default) - required for multi-tenant Loki; batch_wait_duration (duration format, default 5s) - wait time before sending batch; batch_size_bytes (number, default 102400 which is 100KiB) - bytes to accumulate before sending batch. Whichever threshold is reached first between batch_wait_duration and batch_size_bytes triggers batch transmission.

Increase log level to debug in Grafana Helm charts

Edit the `values.yaml` file and locate the `grafana.ini:` section. Under the logging configuration where `mode: console` is set, add `level: debug` on the next line with matching indentation. Run `helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring` to apply the changes. Verify the change in the Grafana UI by navigating to Server Admin > Settings and searching for the log level.

Uninstall Grafana Helm deployment

Run `helm uninstall my-grafana -n monitoring` to delete all objects from the namespace. This removes the Grafana deployment. To also delete the namespace itself, run `kubectl delete namespace monitoring`.

Community-maintained Grafana Helm Charts

The Grafana open source community offers Helm Charts for running Grafana on Kubernetes. The charts are provided without warranties. The code is maintained in the Community-maintained GitHub repository at https://github.com/grafana-community/helm-charts. The values.yaml file can be downloaded from https://github.com/grafana-community/helm-charts/blob/main/charts/grafana/values.yaml.

Reset Grafana admin credentials in Helm deployment

Edit the `values.yaml` file and locate the `adminUser` and `adminPassword` fields. Set them to desired values, for example `adminUser: admin` and `adminPassword: admin`. Run `helm upgrade my-grafana grafana-community/grafana -f values.yaml -n monitoring` to apply the changes. The new credentials will be used for the super admin account.

Helm repository setup commands

To set up the Grafana Helm repository, use `helm repo add <DESIRED-NAME> <HELM-REPO-URL>`. The community-maintained repository is at `helm repo add grafana-community https://grafana-community.github.io/helm-charts`. Verify the repository with `helm repo list` and update to the latest charts with `helm repo update`.

Deploy Grafana Helm chart to Kubernetes namespace

Create a namespace first with `kubectl create namespace monitoring`. Then deploy the Grafana Helm chart with `helm install my-grafana grafana-community/grafana --namespace monitoring`, where `my-grafana` is the logical chart name, `grafana-community/grafana` is the repository and package name, and `--namespace` specifies the target namespace. Verify deployment with `helm list -n monitoring`.

Use separate namespace for Grafana Helm deployment

It is recommended to deploy Grafana Helm charts into a separate namespace instead of the default namespace. This helps avoid conflicts with other applications and allows better organization, allocation, and management of cluster resources. A common practice is to use a namespace named `monitoring`.

Give your agent this brain