If you have an elixir app running on Fly.io, it's easy to send telemetry data to GrafanaCloud.

Lets start with Logs.

Logs

Shipping logs to GrafanaCloud requires setup and deployment of another app called LogShipper.

LogShipper Documentation

Setting up LogShipper

• Create a new directory

mkdir logshipper

• Create a new app, but don't launch yet

fly launch --no-deploy --image ghcr.io/superfly/fly-log-shipper:latest`

• Configure your org and access token

fly secrets set ORG=personal
fly secrets set ACCESS_TOKEN=$(fly auth token)

• Configure your Loki credentials. Find these in the GrafanaCloud Portal, Loki section. Hint: you may have to generate an API key.

fly secrets set LOKI_URL=
fly secrets set LOKI_USERNAME=
fly secrets set LOKI_PASSWORD=

• Add this to the newly generated fly.toml file:

[[services]]
  http_checks = []
  internal_port = 8686

• Deploy

flyctl deploy

Once deployed, this should start sending all the logs from all your fly apps in the configured organization to GrafanaCloud.

Find all configuration options for the LogShipper app in the repository

Also, I'd recommend shipping your logs in JSON format using something like the `logger_json` library.

The `logger_json` library

Traces

To send Traces to GrafanaCloud Tempo, first add the OpenTelemetry libraries. Depending on your needs, you may leave off some of these.

# ./mix.exs
defp deps do
  [
    ...
    {:opentelemetry_exporter, "~> 1.0"},
    {:opentelemetry, "~> 1.0"},
    {:opentelemetry_api, "~> 1.0"},
    {:opentelemetry_ecto, "~> 1.0"},
    {:opentelemetry_liveview, "~> 1.0.0-rc.4"},
    {:opentelemetry_phoenix, "~> 1.0"},
    {:opentelemetry_cowboy, "~> 0.2"}
  ]
end

Now add some configuration to the config/runtime.exs file.

# ./config/runtime.exs
if config_env() == :prod do
  otel_auth = System.get_env("OTEL_AUTH") ||
    raise """
    OTEL_AUTH is a required variable
    """

  config :opentelemetry_exporter,
    otlp_protocol: :grpc,
    otlp_traces_endpoint: System.fetch_env!("OTLP_ENDPOINT"),
    otlp_headers: [{"Authorization", "Basic #{otel_auth}"}]
end

Next, setup the environment variables.

• The value required for `OTLP_ENDPOINT` can be found on the GrafanaCloud Portal Tempo section, and will look something like: `https://tempo-us-central1.grafana.net/tempo` (your URL may differ)
• The value for `OTEL_AUTH` is a base64 encoded value of `{username}:{api token}`.

echo -n 'username:password' | base64`

(replace username and password with the actual values)

• And you'll need the data source name (found on the same GrafanaCloud Portal page) which will be used to set the value of `OTEL_RESOURCE_ATTRIBUTES`

All of these values can be set with one command:

flyctl secrets set OTLP_ENDPOINT=https://your_endpoint OTEL_RESOURCE_ATTRIBUTES=your_datasource_name OTEL_AUTH=your_base64_encoded_string

After setting these values and deploying the application, traces should start showing in Grafana!

Metrics

For Metrics, I really like to use Prometheus and I find the easiest way to get started with Prometheus is using prom_ex. PromEx provides excellent documentation and is worth reading, but a quick guide to get it working:

Prometheus Overview

Documentation for prom_ex

• Add `:prom_ex, "~> 1.8"` to your dependencies in `mix.exs` and run `mix deps.get`
• Run the generator

mix prom_ex.gen.config --datasource curl`

• Add configuration in `config.exs` for the metrics server

config :your_app, YourApp.PromEx,
    metrics_server: [
    port: System.get_env("PROM_PORT") || 9091,
    path: "/metrics",
    protocol: :http,
    pool_size: 5
]

• Add `YourApp.PromEx` to your supervision tree in `application.ex`

def start(_type, _args) do
    children = [
      YourAppWeb.Endpoint,
      # PromEx should be started after the Endpoint, to avoid unnecessary error messages
      YourApp.PromEx,
      ...
    ]

• Lastly, uncomment any desired plugins in the generated `YourApp.PromEx` file.

Now running the app exposes the metrics at `localhost:9091/metrics`.

Next, expose the metrics so that Fly can scrape them. As documented by Fly, just add the following to your fly.toml file:

Fly documentation for metrics

[metrics]
port = 9091
path = "/metrics"

Setup the Prometheus data source in GrafanaCloud with the following properties:

• HTTP -> URL "https://api.fly.io/prometheus/<org-slug>/" (replace `org_slug` with your org)
• Custom HTTP Headers -> + Add Header:
• Header: Authorization, Value: Bearer <token> (replace `token` with the result of `flyctl auth token`

You should now see fly metrics and `prom_ex` defined metrics in GrafanaCloud.

Wrap up

I wrote this because I wanted to add observability to my Elixir/Phoenix apps that run on Fly and finding the information I needed was scattered throughout the docs.

Happy Observing

Syndicate to the Fediverse