provisioning

134 notes in this subject, read out of this brain and free to use. This is page 1 of 3.

Grafana Operator prerequisites for ArgoCD dashboard management

To manage Grafana dashboards with GitOps using ArgoCD and Grafana Operator, you need: an existing Grafana Cloud stack, a Kubernetes cluster with Grafana Operator installed, ArgoCD installed on the Kubernetes cluster, and a Git repository to store dashboard configurations.

ArgoCD directory recurse for nested dashboard configurations

When configuring ArgoCD to sync Grafana dashboards, enable the directory recurse option (either via UI checkbox 'Directory Recurse' or in Application manifest as source.directory.recurse: true) to allow ArgoCD to discover and apply all dashboard Custom Resources in nested subdirectories of the grafana folder.

GrafanaDashboard Custom Resource with inline JSON

Create a GrafanaDashboard Custom Resource with apiVersion grafana.integreatly.org/v1beta1 and kind GrafanaDashboard. Include spec fields: resyncPeriod (e.g. '30s') for synchronization interval, instanceSelector with matchLabels for dashboard matching (matching Grafana CR labels), and json field containing the dashboard JSON configuration. The dashboard JSON must include fields: id (null for new), title, tags, style, timezone, editable, hideControls, graphTooltip, panels, time, timepicker, templating, annotations, refresh, schemaVersion, version, and links.

Grafana Custom Resource YAML structure for Grafana Operator

Configure the Grafana Custom Resource with apiVersion grafana.integreatly.org/v1beta1 and kind Grafana. Metadata includes the name (Grafana Cloud stack name), namespace, and labels with dashboards key matching the stack name. The spec.external section contains the URL (https://<STACK_NAME>.grafana.net/) and apiKey reference to the secret containing GRAFANA_CLOUD_INSTANCE_TOKEN.

ArgoCD Application manifest for Grafana dashboards GitOps

Create an ArgoCD Application resource with apiVersion argoproj.io/v1alpha1 and kind Application. Set metadata.name and namespace. In spec: destination server is 'https://kubernetes.default.svc', source includes repoURL (Git repository URL), path (to grafana folder), targetRevision 'HEAD', and directory.recurse set to true. Set syncPolicy.automated.prune and selfHeal to true, syncOptions with CreateNamespace=true, and retry limit to 2 with backoff duration 5s and maxDuration 3m0s with factor 2.

Grafana API Token Secret YAML structure for Grafana Operator

Create a Kubernetes Secret to store the Grafana API Token with apiVersion v1, kind Secret. The metadata must include a name (typically 'grafana-cloud-credentials') and namespace where grafana-operator is deployed. The stringData field contains the key GRAFANA_CLOUD_INSTANCE_TOKEN with the API key value. Set type to Opaque.

Dashboard update workflow with Grafana Operator and ArgoCD

To update a dashboard: modify the dashboard JSON configuration in the Git repository, commit and push changes, ArgoCD detects the update and synchronizes the changes to the GrafanaDashboard Custom Resource, and Grafana Operator then syncs the changes to the Grafana instance. Changes are reflected in the Grafana UI after synchronization completes.

GrafanaDashboard Custom Resource from grafana.com

To import a dashboard from grafana.com, create a GrafanaDashboard Custom Resource with apiVersion grafana.integreatly.org/v1beta1 and kind GrafanaDashboard. Include spec.instanceSelector with matchLabels for the Grafana instance, and spec.grafanaCom with the dashboard id from grafana.com (e.g., id: 1860).

GrafanaDashboard Custom Resource from ConfigMap reference

To manage a dashboard defined in a ConfigMap, create both a ConfigMap resource and a GrafanaDashboard Custom Resource. The ConfigMap stores the dashboard JSON in its data field. The GrafanaDashboard spec includes configMapRef with name and key fields pointing to the ConfigMap and the specific data key containing the JSON.

ArgoCD automatic sync policy for Grafana dashboards

Set the ArgoCD sync policy to automatic with prune and self-heal enabled. This ensures that any changes to dashboard configurations committed to the Git repository are automatically detected and synchronized to the Grafana instance, and any manual changes in the cluster are reverted to match the Git source of truth.

Grafana API Key creation for Grafana Operator authentication

To authenticate Grafana Operator with Grafana, create an API key in your Grafana instance. Reference the API Key Documentation to generate the key, then store it in the grafana-cloud-credentials Secret as the GRAFANA_CLOUD_INSTANCE_TOKEN value.

Git repository structure for Grafana Operator with ArgoCD

Organize Grafana configurations in a Git repository with a root folder named 'grafana', containing a subdirectory 'dashboards' for dashboard configurations. Store the Grafana API Token secret in 'grafana-token.yml' and Grafana Custom Resource in 'grafana-cloud.yml' at the root of the grafana folder. All dashboard GrafanaDashboard resources go in the dashboards subdirectory.

Notification alerts Terraform configuration example

Example Terraform resource for notification alerts configuration: resource "grafana_asserts_notification_alerts_config" "example" { provider = grafana.asserts name = "ExampleAlert" match_labels = { alertname = "HighCPUUsage" job = "monitoring" } alert_labels = { severity = "warning" team = "platform" } duration = "5m" silenced = false }

grafana_asserts_notification_alerts_config Terraform resource

The grafana_asserts_notification_alerts_config resource manages Knowledge Graph notification alerts configurations through the Grafana API. Arguments: name (string, required) - the name of the notification alerts configuration, immutable and forces recreation if changed; match_labels (map(string), optional) - labels to match for this configuration, used to filter which alerts this configuration applies to; alert_labels (map(string), optional) - labels to add to alerts generated by this configuration; duration (string, optional) - duration for which the condition must be true before firing (for example '5m', '30s'), maps to 'for' in Knowledge Graph API; silenced (bool, optional) - whether this notification alerts configuration is silenced, defaults to false.

Foundation SDK dashboard generation example (TypeScript)

Example TypeScript code showing how to generate a Grafana dashboard using the Foundation SDK and wrap it for Kubernetes-style deployment: ```typescript import { DashboardBuilder, RowBuilder } from '@grafana/grafana-foundation-sdk/dashboard'; import * as fs from 'fs'; // Generate the dashboard JSON const dashboard = new DashboardBuilder('My Dashboard') .uid('my-dashboard') .tags(['generated', 'foundation-sdk', 'typescript']) .refresh('5m') .time({ from: 'now-1h', to: 'now' }) .timezone('browser') .withRow(new RowBuilder('Overview')) .build(); // Convert to Kubernetes-style format const dashboardWrapper = { apiVersion: "dashboard.grafana.app/v1", kind: "Dashboard", metadata: { name: dashboard.uid! }, spec: dashboard }; // Save the formatted JSON to a file const dashboardJSON = JSON.stringify(dashboardWrapper, null, 2); fs.writeFileSync('dashboard.json', dashboardJSON, 'utf8'); console.log(`Dashboard JSON:\n${dashboardJSON}`); ```

Foundation SDK dashboard generation example (Go)

Example Go code showing how to generate a Grafana dashboard using the Foundation SDK and wrap it for Kubernetes-style deployment: ```go package main import ( "encoding/json" "log" "os" "github.com/grafana/grafana-foundation-sdk/go/cog" "github.com/grafana/grafana-foundation-sdk/go/common" "github.com/grafana/grafana-foundation-sdk/go/dashboard" ) type DashboardWrapper struct { APIVersion string `json:"apiVersion"` Kind string `json:"kind"` Metadata Metadata `json:"metadata"` Spec dashboard.Dashboard `json:"spec"` } type Metadata struct { Name string `json:"name"` } func main() { builder := dashboard.NewDashboardBuilder("My Dashboard"). Uid("my-dashboard"). Tags([]string{"generated", "foundation-sdk", "go"}). Refresh("5m"). Time("now-1h", "now"). Timezone(common.TimeZoneBrowser). WithRow(dashboard.NewRowBuilder("Overview")) dashboard, err := builder.Build() if err != nil { log.Fatalf("failed to build dashboard: %v", err) } dashboardWrapper := DashboardWrapper{ APIVersion: "dashboard.grafana.app/v1", Kind: "Dashboard", Metadata: Metadata{ Name: *dashboard.Uid, }, Spec: dashboard, } dashboardJson, err := json.MarshalIndent(dashboardWrapper, "", " ") if err != nil { log.Fatalf("failed to marshal dashboard: %v", err) } err = os.WriteFile("dashboard.json", dashboardJson, 0644) if err != nil { log.Fatalf("failed to write dashboard to file: %v", err) } log.Printf("Dashboard JSON:\n%s", dashboardJson) } ```

Kubernetes-style dashboard JSON wrapper format

To deploy dashboards using the Foundation SDK with Grafana's Kubernetes resource compatible API, wrap the dashboard JSON in a format with apiVersion set to 'dashboard.grafana.app/v1', kind set to 'Dashboard', metadata containing a name field (typically the dashboard UID), and spec containing the full dashboard object.

Benefits of automating dashboard provisioning with CI/CD

Automating Grafana dashboard deployment ensures dashboards remain consistent across environments, provides full version control to track changes and roll back if needed, prevents dashboard duplication through intelligent checks before create or update operations, eliminates manual JSON file uploads, and allows teams to focus on improving dashboards rather than managing deployment.

GitHub Actions workflow for dashboard deployment with gcx

GitHub Actions workflow configuration for automating Grafana dashboard deployment using the Foundation SDK and gcx CLI tool. The workflow: 1. Triggers on push to main branch 2. Sets up Go 1.24.6 3. Downloads and installs gcx CLI from GitHub 4. Runs the dashboard generator to produce dashboard.json 5. Deploys the dashboard using: `gcx resources push dashboards --path ./dashboard.json` 6. Uses environment variables: GRAFANA_SERVER, GRAFANA_STACK_ID, GRAFANA_TOKEN 7. Requires GitHub variables: vars.GCX_VERSION, vars.GRAFANA_SERVER, vars.GRAFANA_STACK_ID 8. Requires GitHub secrets: secrets.GRAFANA_TOKEN (service account token with sufficient permissions) The workflow checks that dashboard.json exists before deployment and exits with error code 1 if it does not.

Git Sync network traffic patterns

Git Sync uses two types of network traffic: sync operations (pull and push) with egress traffic from Hosted Grafana IPs to the Git server (customer Git servers must allow inbound traffic from these IPs), and webhooks (instantaneous sync) with inbound traffic to the stack's public endpoint requiring the Git server to reach *.grafana.net.

Git Sync provisioning feature toggle

The provisioning feature toggle is enabled by default in Grafana Cloud and, starting in Grafana v13, for OSS and Enterprise. No manual configuration is required.

GitHub App creation and configuration steps

To create a GitHub App for Git Sync: (1) Go to https://github.com/settings/apps/new, (2) Fill in a unique Name and Homepage URL (e.g., your Grafana Cloud instance URL), (3) Uncheck the Active box in the Webhook section, (4) In Repository permissions set Administration to Read-only, Contents to Read and write, Metadata to Read-only, Pull requests to Read and write, and Webhooks to Read and write, (5) Select 'Only on this account' under 'Where can this GitHub App be installed?', (6) Click Create Github App.

Git Sync prerequisites

Before setting up Git Sync, you must have: a Grafana instance (Cloud, OSS, or Enterprise), administration rights in your Grafana organization, and a compatible Git provider. If using webhooks or image rendering, you need a public instance with external access. Optionally, the Image Renderer service can save image previews with pull requests.

Enable Git providers in configuration

For Grafana Enterprise v12.4.0 or Grafana OSS v12.4.0 using pure Git, GitLab, or Bitbucket, add the following to your Grafana configuration file (grafana.ini or custom.ini) under the [provisioning] section: repository_types = "git|github|bitbucket|gitlab|local". After adding this configuration, restart Grafana.

GitHub App installation and Installation ID retrieval

To install a GitHub App: (1) Click Install App at the top left of the App page, (2) Choose the user for installation and select repositories, (3) Click Install, (4) Copy the installationID from the page URL https://github.com/settings/installations/installationID.

GitHub App parameters required for authentication

To authenticate Git Sync with a GitHub App, you need: GitHub App ID, GitHub App Private Key, and GitHub App Installation ID.

Default folder roles in Git Sync

By default, folders provisioned with Git Sync have these role-to-role mappings: Admin = Admin, Editor = Editor, Viewer = Viewer.

Allow internal or private Git servers with allowed_git_urls

By default, Git Sync rejects repository URLs with hosts resolving to loopback, private (RFC 1918), link-local, or unspecified addresses to protect against SSRF. For Grafana v13.0.4 and v13.1.1 and later (self-managed OSS and Enterprise only), to connect to a private Git server, add hosts to the allowed_git_urls allowlist in the [provisioning] section of your configuration file as a comma-separated list: allowed_git_urls = git.internal.example.com, ghe.example.com:8443. Each entry can be a hostname, host:port, a full URL (only the host is used), a literal IP address, or a CIDR range.

GitHub App ID and private key retrieval

After creating a GitHub App, copy the AppID from the About section. Generate a private key from the banner or by scrolling to the Private Keys section; a PEM file containing your private key will be downloaded to your computer.

Git Sync does not use AWS PrivateLink or PDC

Git Sync does not route over AWS PrivateLink or Private Data Source Connect (PDC). It uses the normal public path from Hosted Grafana IPs and is independent of PrivateLink/PDC. If you use AWS PrivateLink or PDC for data sources, you can still use Git Sync as the two features neither interfere with nor depend on each other.

Git Sync direct commit mode

When the repository allows writes and branch protection is not enabled, Git Sync commits dashboard changes directly to the configured branch without review. Use this mode when rapid iteration is needed and changes don't require formal review, such as in development environments.

Git Sync authentication required permissions

Git Sync authentication credentials must have specific permissions at the Git provider. Required for all configurations: read access to repository contents and read access to branch information. Required for writing changes: write access to create commits, permission to create pull requests (when branch protection is enabled), and permission to push to feature branches (for creating pull requests). Optional for instant synchronization: permission to create and manage webhooks.

Git Sync pull requests not created troubleshooting

When pull requests are not created when expected, the cause is either that branch protection is not enabled or the authentication credentials lack pull request creation permission. To resolve: verify branch protection is enabled on the correct branch, check that the credentials have permission to create pull requests, and ensure the branch name in Git Sync settings matches the protected branch exactly.

Git Sync commits directly without review troubleshooting

When dashboard changes commit directly without review, the cause is that branch protection is not configured at the Git provider. To resolve: enable branch protection on the target branch at the Git provider, configure the branch to require pull requests before merging, and verify the branch name in protection rules matches the branch configured in Grafana.

CODEOWNERS files assign reviewers for Git Sync pull requests

Many Git providers support CODEOWNERS files that automatically assign reviewers to pull requests based on which files are changed. When Git Sync creates a pull request, the Git provider uses the CODEOWNERS file to assign the appropriate team or users for review. This ensures dashboard changes are reviewed by the teams responsible for those dashboards, based on folder path or file patterns.

Branch protection rules enforce Git Sync pull request creation

Branch protection rules at the Git provider enforce how changes are made to specific branches. When enabled on the branch that Git Sync targets, these rules require Git Sync to create pull requests instead of pushing commits directly. Common use cases include production environments requiring change approval, compliance requirements for audit trails and review, and multi-team environments where changes need visibility. Branch protection can enforce various controls such as requiring pull requests before merging, setting reviewer approval requirements, running automated validation checks, preventing force pushes, and restricting who can push directly to protected branches.

Git Sync pull request mode

When branch protection is enabled at the Git provider, Git Sync creates pull requests instead of committing directly. Changes require review and approval before merging to the main branch. Use this mode when changes require review and approval, such as in production environments or when multiple teams collaborate on dashboards.

Git Sync read-only mode

Configure the repository as read-only in Grafana to prevent any writes to Git from the Grafana UI. Dashboards sync from Git to Grafana, but users cannot save changes back to Git. Use this mode when Git is the single source of truth and all changes must be made through direct Git commits or CI/CD processes.

Direct write access vs protected branches in Git

Direct write access allows users with write permission to push commits directly to branches. For Git Sync to push dashboard changes from Grafana to Git, authentication credentials must have write access to the repository. Protected branches can restrict direct writes and require changes to go through pull requests with review and approval, even for users with write access.

Public vs private Git repositories for Git Sync

Public repositories allow anyone to view repository contents including dashboard configurations and any data or queries they contain; only use public repositories if dashboards contain no sensitive information. Private repositories restrict visibility to authorized users only, protecting dashboard configurations, queries, and any embedded credentials or sensitive data from public access. For Git Sync to function, authentication credentials must have read access to pull dashboard changes from Git to Grafana.

Git Sync 403 Forbidden error troubleshooting

When Git Sync fails with 403 Forbidden or Unauthorized errors, the cause is that authentication credentials lack the required repository permissions. To resolve: verify the credentials have read and write access to the repository, check that the credentials can create pull requests (if branch protection is enabled), verify authentication credentials haven't expired, and for GitHub Apps verify the app is installed and authorized for the repository.

Git Sync protects dashboard source code through two access points

Dashboard source code can be accessed through two paths: Grafana files endpoint (users can view and edit through the Grafana files API endpoint, controlled by Grafana folder and dashboard permissions) and Git repository (users with repository access can view and modify dashboard files directly in Git, controlled by Git provider permissions). Both access points must be protected to secure dashboard configurations.

Git Sync primary-replica how it works

In a primary-replica deployment: (1) Both instances stay synchronized through Git. (2) Reverse proxy routes traffic to primary. (3) Users edit on primary, Git Sync commits changes. (4) Both instances pull latest changes to keep replica in sync. (5) On primary failure, proxy fails over to replica.

Git Sync primary-replica failover considerations

For a primary-replica Git Sync deployment, consider health checks and monitoring, continuous syncing to minimize data loss, and planning failback procedures (automatic or manual).

Git Sync load balancer how it works

In a load balancer deployment: (1) All instances stay synchronized through Git. (2) Load balancer distributes incoming traffic across all active instances. (3) Users can view dashboards from any instance. (4) When a user modifies a dashboard on any instance, Git Sync commits the change. (5) All other instances pull the updated dashboard during their next sync cycle, or instantly if webhooks are configured. (6) If one instance fails, load balancer stops routing traffic to it and remaining instances continue serving.

Git Sync load balancer important considerations

For a load balancer Git Sync deployment: Instances are eventually consistent due to sync intervals, so instances may briefly have different dashboard versions. Multiple users editing the same dashboard on different instances can cause conflicts. Instances should share the same backend database for user sessions, preferences, and annotations. Design for stateless operation where possible to maximize load balancing effectiveness.

Git Sync load balancer use cases

Use a load balancer Git Sync scenario to handle significant user load (high traffic), to distribute user requests across multiple instances, to ensure service continuity during maintenance or failures (maximum availability), to add instances as load increases (scalability), and to provide fast response times under heavy load (performance).

Git Sync primary-replica use cases

Use a primary-replica Git Sync scenario for automatic failover when the primary instance fails, to guarantee dashboard availability for high availability requirements, to implement high availability simply without active-active complexity, to perform maintenance updates while another instance serves traffic, and to ensure dashboard access does not tolerate downtime.

Git Sync load balancer configuration parameters

In a load balancer Git Sync setup, all instances use identical configuration: Repository (e.g., your-org/grafana-manifests), Branch (e.g., main), and Path (e.g., shared/). All instances sync from the same path in the same repository and branch.

Git Sync primary-replica configuration parameters

In a primary-replica Git Sync setup, both master and replica instances use identical configuration: Repository (e.g., your-org/grafana-manifests), Branch (e.g., main), and Path (e.g., shared/). Both instances sync from the same path in the same repository and branch.

Git Sync load balancer active-active scenario

In a load balancer Git Sync deployment, multiple active Grafana instances run behind a load balancer that distributes requests across them using round-robin or similar algorithms. All instances sync from the same Git repository path (e.g., shared/). Any instance can serve read requests and accept dashboard modifications. When a user modifies a dashboard on any instance, Git Sync commits the change, and other instances pull the updated dashboard during their next sync cycle or instantly if webhooks are configured. All instances show identical folder structure and dashboards.

Git Sync primary-replica high availability scenario

In a primary-replica Git Sync deployment, a master Grafana instance (active) and one or more replica instances (standby) synchronize with the same Git repository location. A reverse proxy routes traffic to the master instance. Both instances pull from the same Git path (e.g., shared/). When the master fails, the proxy automatically fails over to a replica. Users see identical folder structure and dashboards regardless of which instance serves traffic. Replicas stay synchronized continuously through Git to minimize data loss.

On-prem file provisioning availability

On-prem file provisioning is available in Grafana v12 and later for both open source and Enterprise editions. It is not available in Grafana Cloud. This feature is only available for dashboards and does not replace classic provisioning at this time.

Provisioned dashboards folder location

After setting up file provisioning, dashboards saved in a GitHub repository or local folder appear in Grafana within a folder named 'provisioned'. Dashboards and folders saved to the local path are referred to as provisioned resources and are labeled as such in the Grafana UI.

On-prem file provisioning purpose

On-prem local file provisioning allows you to add resources stored in your local file system to your Grafana instance. You can configure how to save dashboards' JSON and other files from your local file system into a single or multiple folders in a different repository, with support for up to 10 connections.

Provisioned resources modification restrictions

Provisioned dashboards can only be modified locally in the provisioned files. Changes made in the provisioned files are reflected in the Grafana database and update the UI to reflect these changes. You cannot use the Grafana UI to edit or delete provisioned resources.

Git Sync usage tier limits table

Git Sync usage limits vary by deployment tier. Cloud Free tier: 1 repository, 20 resources per repository. Cloud Other tiers: 10 repositories, 1,000 resources per repository. On-prem OSS: 10 repositories (default), no resource limit. On-prem Enterprise: 10 repositories (default), no resource limit. The 10-repository default on self-managed Grafana is not a hard ceiling and can be raised with the [provisioning] max_repositories configuration setting (0 means unlimited). Resources per repository are unlimited by default on-prem (max_resources_per_repository = 0). Grafana Cloud tier limits cannot be changed from configuration.

Git Sync performance impact with many folders

When Git Sync is enabled, database load might increase, especially if your Grafana instance has many folders and nested folders. You should evaluate the performance impact in a non-production environment before enabling Git Sync in production.

Git Sync 1000 resources per repository recommendation

Do not sync more than 1,000 resources per repository connection. Beyond roughly 1,000 resources per connection, the sync workflow puts noticeable load on Grafana itself, which may result in slower syncs and increased database load. This is a performance recommendation, not a hard cap on self-managed Grafana, but it is a tier limit on Grafana Cloud.

Git Sync max_repositories configuration setting

The max_repositories configuration setting controls how many repositories you can sync on self-managed Grafana. The default is 10, but you can set it to 0 for unlimited repositories. This setting is located in the [provisioning] section of the Grafana configuration file.

Give your agent this brain