godeltaprof endpoints for continuous profiling
Additional godeltaprof endpoints are available for continuous profiling scenarios: `/debug/pprof/delta_heap`, `/debug/pprof/delta_block`, and `/debug/pprof/delta_mutex`.
Grafana dashboards · all subjects
overview
262 notes in this subject, read out of this brain and free to use. This is page 4 of 5.
Profiling environment variables
Configure profiling settings using these environment variables: `GF_DIAGNOSTICS_PROFILING_ENABLED` (boolean), `GF_DIAGNOSTICS_PROFILING_ADDR` (address), `GF_DIAGNOSTICS_PROFILING_PORT` (port number), `GF_DIAGNOSTICS_PROFILING_BLOCK_RATE` (fraction), and `GF_DIAGNOSTICS_PROFILING_MUTEX_RATE` (fraction).
pprof debugging endpoints location
pprof debugging endpoints are served on a different port than the Grafana HTTP server. Browse to `http://<profile-addr>:<profile-port>/debug/pprof` to check available debugging endpoints.
Collect heap profiles for memory leak analysis
To analyze high memory usage or memory leaks, collect multiple heap profiles over time with delays between collections: `curl http://<profile-addr>:<profile-port>/debug/pprof/heap > heap1.pprof`, then wait (e.g., 30 seconds), then `curl http://<profile-addr>:<profile-port>/debug/pprof/heap > heap2.pprof`. Compare profiles using: `go tool pprof -http=localhost:8081 --base heap1.pprof heap2.pprof`.
Collect CPU profiles for high CPU analysis
When experiencing high CPU usage, collect CPU profiles over a period of time (e.g., 30 seconds) using: `curl 'http://<profile-addr>:<profile-port>/debug/pprof/profile?seconds=30' > profile.pprof`. Analyze with: `go tool pprof -http=localhost:8081 profile.pprof`.
Enable tracing with command-line flags
Start grafana-server with the `-tracing` flag to enable tracing and `-tracing-file` to override the default trace file (trace.out) where the trace result is written.
Usage insights logs feature availability
Usage insights log export is available in Grafana Enterprise and to customers on select Grafana Cloud plans.
Usage insights logs trigger events
A log is created every time a user opens a dashboard, a query is sent to a data source in the dashboard view, or a query is performed via Explore.
Usage insights log fields reference
Usage insights logs contain the following fields: eventName (string, always present) - type of event (data-request or dashboard-view); folderName (string, always present) - name of dashboard folder; dashboardName (string, always present) - name of the dashboard; dashboardId (number, always present) - ID of the dashboard; datasourceName (string) - name of queried data source; datasourceType (string) - type of data source (e.g. prometheus, elasticsearch, loki); datasourceId (number) - ID of queried data source; panelId (number) - ID of the query panel; panelName (string) - name of the query panel; error (string) - error returned by query; duration (number) - duration of query; source (string) - source of query (e.g. dashboard or explore); orgId (number, always present) - ID of user's organization; orgName (string, always present) - name of user's organization; timestamp (string, always present) - request date/time in UTC RFC3339 format; tokenId (number, always present) - ID of user's authentication token; username (string, always present) - name of Grafana user making request; userId (number, always present) - ID of Grafana user making request; totalQueries (number, always present) - number of queries executed for data request; cachedQueries (number, always present) - number of cached queries fetched.
Usage insights dashboard visualization
Grafana provides pre-built dashboards for visualizing usage insights exported to Loki: Usage insights dashboard (ID 13785) and Usage insights datasource details dashboard (ID 13786). These can be imported to understand Grafana instance usage patterns.
Windows installation methods
Grafana on Windows can be installed using either the Windows installer or the standalone Windows binary file.
Windows installer download and install steps
To install Grafana using the Windows installer: navigate to the Grafana download page, select the desired version (most recent selected by default), select an Edition (Enterprise or Open Source), click Windows, then click Download the installer. Open and run the downloaded installer.
Windows standalone binary download and setup steps
To install Grafana using the standalone Windows binary: navigate to the Grafana download page, select the desired version, select an Edition, click Windows, then click Download the zip file. Right-click the downloaded file, select Properties, check the unblock checkbox, and click OK. Extract the ZIP file to any folder.
Running Grafana on Windows
Execute grafana-server.exe located in the bin directory, preferably from the command line. Access Grafana by opening a browser and navigating to the Grafana port (http://localhost:3000/ by default).
Running Grafana as a Windows service
To run Grafana as a Windows service, download NSSM (a service manager tool) and use it to add Grafana as a Windows service.
Default Grafana port on Windows
The default Grafana port is 3000 (http://localhost:3000/). This port might require extra permissions on Windows. Port 8080 does not require extra Windows privileges and can be used as an alternative.
Changing Grafana port on Windows
To change the Grafana port on Windows: open the conf directory and copy sample.ini to custom.ini. Edit custom.ini (never defaults.ini) and uncomment the http_port configuration option by removing the leading semicolon. Change the port value to the desired port number such as 8080.
Enterprise vs Open Source edition on Windows
Enterprise is the recommended Grafana edition and is functionally identical to the Open Source version but includes features that can be unlocked with a license. The Open Source version is functionally identical to Enterprise but requires downloading the Enterprise version if Enterprise features are desired.
Installing nightly Grafana builds on Windows
To install a nightly build of Grafana on Windows instead of a tagged release, click Nightly Builds on the Grafana download page and then select a version.
Default Grafana login credentials
The default username and password for the first sign in to Grafana are both 'admin'.
Default Grafana URL
Unless configured differently, Grafana is set to use http://localhost:3000 by default as the root URL.
First login password change prompt
After signing in with the default admin credentials, Grafana prompts the user to change the password. The user must click OK on the prompt and enter a new password.
Recommendation to change default administrator password
It is strongly recommended to change the default administrator password after the first sign in to Grafana.
Authentication methods overview
Grafana supports numerous authentication methods to verify user identity. The authentication configuration dictates which users can access Grafana and the methods they can use for logging in.
Automatic user role and team membership sync from auth provider
Grafana can be configured to automatically update users' roles and team memberships in Grafana based on the information returned by the auth provider integration.
Considerations for choosing authentication method
When deciding on an authentication method, it is important to take into account your current identity and access management system as well as the specific authentication and authorization features you require.
Navigate to organization preferences
To access organization preferences in Grafana, hover your cursor over the Configuration (gear) icon, then click Preferences.
Data source plugin role
Data source plugins take a query you want answered, retrieve the data from the data source, and reconcile the differences between the data model of the data source and the data model of Grafana dashboards.
Dashboard structure: panels, rows, and tabs
A Grafana dashboard is a set of one or more panels, organized and arranged into one or more rows or tabs, that provide an at-a-glance view of related information. Panels are created using components that query and transform raw data from a data source into visualizations.
Data source types supported
A data source can be an SQL database, Grafana Loki, Grafana Mimir, or an API endpoint. It can even be a basic CSV file. Grafana has 150+ data source plugins available to unify all data sources into a single dashboard.
Query purpose in dashboards
Queries allow you to reduce the entirety of your data to a specific dataset, providing a more manageable visualization. Since data sources have their own distinct query languages, Grafana dashboards provide a query editor to accommodate these differences.
Transformation purpose
When the data format in a visualization doesn't meet your requirements, you can apply a transformation that manipulates the data returned by a query.
Grafana server-side rendering with phantomjs
Grafana 2.0 and later includes server-side rendering capability using phantomjs that can render any graph or panel to a PNG image. This feature works across all data sources and produces PNG images that look identical to the browser rendering. The phantomjs binary ships with Grafana on Linux. To verify this feature works, use the 'Direct link to rendered image' link in the panel share dialog. For phantomjs to work properly, required font packages must be installed on the server.
Folder deletion is immediate and irreversible
When you delete folders, the action is immediate and cannot be reversed.
Deleted dashboards can be restored
Deleted dashboards are stored in the deletion history and can be restored if needed, unlike deleted folders.
Dashboard deletion history retention
Deleted dashboards are stored in the deletion history for up to 12 months. The deletion history can contain a maximum of 1000 dashboards. After this limit is reached, the oldest deleted dashboards are permanently removed even if the 12-month expiration period isn't over yet.
Access deleted dashboards list
You can access the list of deleted dashboards from the Dashboards page by clicking the Recently deleted button, or by navigating to Dashboards > Recently deleted.
Restore deleted dashboards permissions
You can restore dashboards that you deleted. Users with admin rights can restore any dashboard.
Restore dashboards procedure
To restore one or more dashboards, navigate to Dashboards > Recently deleted, select the dashboards you want to restore, click Restore, and select a folder to which the dashboards will be moved. If you select one dashboard and its original folder still exists, that folder is preselected. If you select multiple dashboards from the same folder, that folder is preselected. If you select dashboards from different folders, you'll need to choose one target folder for all of them. If the original folder no longer exists, you'll need to select a new target folder. Then click Restore.
Dashboard restoration limitations: permissions not preserved
Dashboard-specific permissions are not restored when you restore a deleted dashboard. After restoration, you'll need to manually reconfigure any custom permissions that were previously set on the dashboard.
Dashboard restoration limitations: folder-level permissions apply
Restored dashboards inherit the permissions of the target folder you select during restoration.
Dashboard restoration limitations: version history is reset
The dashboard's version history is not preserved when restoring a deleted dashboard. After restoration, the dashboard starts with version 1, and all previous versions are lost.
Generative AI features for dashboards
Grafana supports generative AI features for dashboards, including generating panel and dashboard titles and descriptions based on data, and generating a summary of changes made to a dashboard when you save it. These features require installation and configuration of Grafana's Large Language Model (LLM) app plugin.
Auto generate option in dashboards
When generative AI features are enabled, an Auto generate option (✨) displays next to the Title and Description fields in panels and dashboards, or when you press the Save button.
Dashboard management tasks available on Dashboards page
On the Dashboards page, you can browse and create dashboard folders, work with team folders, manage folder permissions, and add generative AI features to dashboards.
Browse dashboards grouping and display
The Dashboards page lists all dashboards you have access to, grouped into folders. Dashboards without a folder are displayed at the top level alongside folders.
Shared with me section in Dashboards
The Shared with me section displays folders and dashboards that are directly shared with you. These aren't shown in the main list because you don't have access to one or more of their parent folders. If you have permission to view all folders, you won't see a Shared with me section.
My team folders section
If you belong to teams that own folders, the Dashboards page displays a My team folders section near the top of the page.
Create a dashboard folder
To create a dashboard folder, click Dashboards in the primary menu, then click New and select New folder in the drop-down. Alternatively, click an existing folder and on the folder's page, click New and select New folder. Enter a unique name and click Create. You can nest folders up to four levels deep.
Edit folder name
To edit a folder name, navigate to the folder from the Dashboards page, click the Edit title icon (pencil) in the header, and update the name. The new folder name is automatically saved.
Folder naming restrictions
Folder names cannot include underscores (_) or percentage signs (%), as these interfere with search functionality. Alerts cannot be placed in folders with slashes (/) in the name, so don't use slashes in folder names if you plan to place alerts in the folder.
Zoom in time range controls
Click and drag horizontally in the panel graph area to select a time range (time series family visualizations only). Type the t+ keyboard shortcut to zoom in.
Dashboard keyboard shortcuts
Grafana dashboard keyboard shortcuts: Ctrl+S saves the current dashboard. f opens the dashboard finder / search. d+k toggles kiosk mode (hides the menu). d+e expands all rows. d+s opens dashboard settings. Ctrl+K opens the command palette. Esc exits panel when in full screen view or edit mode, and returns to the dashboard from dashboard settings.
Focused panel keyboard shortcuts
When hovering over a panel with the mouse, the following shortcuts target that panel: e toggles panel edit view, v toggles panel full screen view, pu copies panel share link, pe opens share panel embed configuration, ps opens share panel snapshot configuration, pd duplicates the panel, pr removes the panel.
Kiosk mode toggle
Kiosk mode hides the main menu and top navigation bar of a dashboard. To turn on kiosk mode, click your user icon and select Enable kiosk mode. To turn off kiosk mode, press Esc.
Time units and relative range syntax
Grafana supports time units: s (seconds), m (minutes), h (hours), d (days), w (weeks), M (months), Q (quarters), and y (years). The minus operator steps back in time relative to now. Append /<time unit> to display the full period of the unit. Use fQ (fiscal quarter) and fy (fiscal year) for fiscal periods. The plus operator steps forward in time relative to now.
Example relative time range values
Last 5 minutes: now-5m to now. The day so far: now/d to now. This week: now/w to now/w. This week so far: now/w to now. This month: now/M to now/M. This month so far: now/M to now. Previous Month: now-1M/M to now-1M/M. This year so far: now/Y to now. This Year: now/Y to now/Y. Previous fiscal year: now-1y/fy to now-1y/fy.
Set absolute time range
To set an absolute time range, type values into the From and To fields with exact time values or relative values such as now-24h, then click Apply time range. Alternatively, click in the From or To field to display a calendar, select the day or days, and click Apply time range. Recently used absolute ranges are displayed.
Semi-relative time range
A semi-relative time range sets the start time to an absolute timestamp and the end time to a relative now. For example, Start time: 2023-05-01 00:00:00, End time: now. To track progress during business hours starting at 8am, set Start time: now/d+8h, End time: now, which is equivalent to Today so far but starts at 8:00am instead of 12:00am. As time progresses, the dashboard automatically zooms out to show more history.
Copy and paste time range
Click the Copy time range to clipboard icon to copy the current time range to the clipboard. Paste the time range into Explore or another dashboard. Keyboard shortcuts t+c and t+v copy and paste time ranges respectively.