Monitor an AI App

 

SigNoz OpenTelemetry Tutorial: Monitor an AI App

AI applications can look very simple from the outside.

You type a question. You wait for a few seconds. An answer appears.

But behind that answer, the application may call an AI model, search a database, use an external tool, read a document, and format the final response.

When one of those steps becomes slow or fails, you need to know exactly what happened.

That is where SigNoz and OpenTelemetry can help.

In this SigNoz OpenTelemetry tutorial, I will explain how to monitor a simple AI application using traces, metrics, logs, dashboards, and alerts. I will also explain why distributed tracing is the SigNoz feature that stood out to me the most.

The simple idea: OpenTelemetry collects information from your application. SigNoz turns that information into useful dashboards, searches, traces, and alerts.

Why I Started Exploring SigNoz

I discovered SigNoz through the Agents of SigNoz hackathon by WeMakeDevs.

The pre-event blog challenge asks participants to self-host SigNoz, connect it as an observability backend, send data, explore its features, and write about the feature they like most. The current blog submission deadline is July 19, 2026.

At first, the word observability sounded complicated.

After breaking it down, I understood that observability simply means being able to see what is happening inside an application.

For example:

• Why did a request fail?
• Why did the chatbot take eight seconds to answer?
• Which API call was slow?
• How often are errors happening?
• Is the server using too much memory?
• Which part of the AI workflow costs the most?

Without observability, developers often have to guess. With observability, they can use real evidence.

What Is SigNoz?

SigNoz is an open-source observability platform powered by OpenTelemetry.

It brings logs, metrics, traces, exceptions, dashboards, and alerts into one platform. Developers can use these tools to monitor applications and investigate problems.

Here is an easy way to understand it.

Imagine your application is a car.

You can see the car moving, but you also need a dashboard to know:

• How fast it is moving
• Whether the engine is overheating
• How much fuel remains
• Whether something is broken

SigNoz is like that dashboard for your application.

It does not build the application for you. It watches the application and helps you understand how well it is working.

What Is OpenTelemetry?

OpenTelemetry, often shortened to OTel, is an open-source system for creating, collecting, and sending telemetry data.

Telemetry is information produced by a running application.

A simple way to remember the difference is:

• OpenTelemetry collects and sends the data.
• SigNoz stores, searches, and displays the data.

OpenTelemetry does not normally create the monitoring dashboard itself. A backend such as SigNoz receives the information and turns it into traces, charts, searches, and alerts.

You can think of OpenTelemetry as a delivery person and SigNoz as the control room.

Logs, Metrics, and Traces Explained Simply

The three main observability signals are logs, metrics, and traces.

Each one answers a different question.

Logs: What happened?

Logs are written records created by an application.

For example:

User question received
AI model request started
Database connection failed
Response returned successfully
Invalid API key

Logs are like your application’s diary.

They help answer:

What exactly happened at this moment?

Logs are especially useful when you need to read an error message or understand a specific event.

Metrics: How is the application performing?

Metrics are numbers measured over time.

Examples include:

• Number of requests
• Average response time
• Error rate
• CPU usage
• Memory usage
• Requests per second
• Number of AI model calls

Metrics are like your application’s scoreboard.

They help answer:

Is the application becoming slower or less reliable?

For example, one slow request may not be a serious problem. But if the average response time keeps increasing, a metric can reveal that pattern.

Traces: Where did the request go?

A trace shows the complete journey of one request.

For example:

Receive user question        15 ms
Search database             180 ms
Call AI model             2,800 ms
Format response              35 ms
Return answer                10 ms

A trace helps answer:

Which step caused the delay or error?

OpenTelemetry describes traces as the path of a request, metrics as measurements captured while a system runs, and logs as records of events.

What I Monitored

For this tutorial, the demo project is a small AI chatbot.

The chatbot accepts a question, sends it to an AI model, and returns the answer. OpenTelemetry watches the request and sends observability data to SigNoz.

The basic architecture looks like this:

User
  ↓
AI chatbot
  ↓
AI model or external tool

At the same time:

AI chatbot
  ↓
OpenTelemetry
  ↓
SigNoz Collector
  ↓
Traces, metrics, logs, dashboards and alerts

You do not need a huge AI system to learn SigNoz.

A small chatbot, Python API, Node.js server, document assistant, or n8n workflow is enough.

Starting small makes it easier to understand what each observability feature is showing.

How to Self-Host SigNoz

SigNoz can be used as a hosted service or installed on your own infrastructure.

For this challenge, I focused on the self-hosted version.

SigNoz currently provides installation options for Docker, Kubernetes, Linux binaries, and several hosting platforms. For a beginner testing on a computer, the standalone Docker option is usually the most direct place to start. Follow the current official installation guide because commands and requirements may change.

The general process is:

1. Install Docker and the required Docker tools.
2. Install the self-hosted SigNoz services.
3. Wait for all containers to start.
4. Open the SigNoz interface in your browser.
5. Check that the services are healthy.

Add screenshot here: Your working SigNoz home screen.

Suggested caption:

My self-hosted SigNoz instance running successfully.

Use your own screenshot. It shows readers and hackathon judges that you actually completed the setup.

Connecting the AI Application to SigNoz

Once SigNoz is running, the next step is to connect the application using OpenTelemetry.

Self-hosted SigNoz accepts telemetry using the OpenTelemetry Protocol, also called OTLP.

The default endpoints are:

• Port 4317 for OTLP over gRPC
• Port 4318 for OTLP over HTTP

SigNoz’s official self-hosted ingestion documentation lists these ports and explains how applications can send telemetry directly to the included OpenTelemetry Collector.

A basic HTTP configuration can look like this:

export OTEL_SERVICE_NAME="ai-chatbot"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"

The service name helps you identify the application inside SigNoz.

The exact endpoint depends on where your application is running.

For example:

• Use localhost when the application and SigNoz are running directly on the same machine.
• Use the collector’s Docker hostname when both are running inside a shared Docker network.
• Use the server address when SigNoz is running on another machine.

After adding the correct OpenTelemetry packages for your programming language, start the application and send a few test requests.

For a Python application, SigNoz documents automatic OpenTelemetry instrumentation for frameworks such as FastAPI, Flask, and Django.

A simple Python setup begins with:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

You can then run the application through OpenTelemetry instrumentation.

The exact run command depends on your framework and file name, so use the official guide that matches your application.

How I Created Useful Test Data

An empty dashboard does not teach you much.

To understand SigNoz, generate different types of requests:

1. Send several successful requests.
2. Send a request that creates a harmless error.
3. Add a short delay to one operation.
4. Call an external API or database.
5. Repeat the requests several times.

This creates different signals to explore.

For example, you might temporarily add a two-second delay to a test endpoint:

import time

def slow_test_operation():
    time.sleep(2)
    return "Slow test completed"

Only use this in a local test environment. Remove the artificial delay after collecting your demonstration data.

Now SigNoz should have something interesting to show.

My Favorite SigNoz Feature: Distributed Tracing

The feature that stood out to me most was distributed tracing.

Why?

Because traces turn a complicated request into a clear timeline.

Without a trace, I may only know:

The chatbot took four seconds to answer.

With a trace, I can see something like:

Chat endpoint              4.1 seconds
├── Validate request       0.02 seconds
├── Search documents       0.30 seconds
├── Call AI model          3.70 seconds
└── Format response        0.08 seconds

Now the problem is much easier to understand.

The application code was not taking most of the time. The AI model call was.

A trace is divided into smaller operations called spans.

Each span can contain information such as:

• Operation name
• Start time
• Duration
• Success or error status
• Service name
• Request attributes
• Related trace information

SigNoz provides tools such as Trace Explorer for examining request paths, while its unified platform can connect traces with logs and metrics.

Distributed tracing replaces “I think this is slow” with “this exact step took 3.7 seconds.”

That is why tracing is so useful for AI applications.

One user request can cause multiple model calls, database searches, tool calls, and API requests. A trace lets you follow those steps in order.

Add screenshot here: A SigNoz trace showing all spans in one chatbot request.

Suggested alt text:

SigNoz distributed trace showing each step of an AI chatbot request

Debugging a Slow AI Request

Here is a simple debugging process you can test in your project.

Step 1: Notice the problem

The chatbot response feels slow.

The user only sees a loading symbol, so the cause is not obvious.

Step 2: Open the trace

Find the slow request inside SigNoz and open its trace.

Check the duration of each span.

Step 3: Find the slowest span

Suppose the trace shows:

Database search       200 ms
AI model call       3,400 ms
Response formatting    40 ms

The AI model call is taking most of the time.

Step 4: Make an improvement

Possible improvements might include:

• Sending a smaller prompt
• Reducing unnecessary model calls
• Using caching where appropriate
• Removing a repeated tool call
• Running independent operations at the same time
• Choosing a faster model for simple tasks

Step 5: Compare the new trace

Run the same test again and compare the result.

Use your real measurements in a table:

Measurement	Before	After
Total response time	[Your result]	[Your result]
AI model call	[Your result]	[Your result]
Number of tool calls	[Your result]	[Your result]
Errors	[Your result]	[Your result]

Do not invent these values. Real measurements make the blog more trustworthy.

Using Metrics to See the Bigger Picture

A trace explains one request.

Metrics help you understand many requests over time.

For my beginner dashboard, I would include:

• Total request count
• Average response time
• Error rate
• Slowest endpoint
• AI model call duration
• Successful versus failed requests

SigNoz supports custom dashboards that can combine information from metrics, logs, and trace data. It also includes application performance charts for measurements such as latency and error rate.

Add screenshot here: Your custom AI application dashboard.

Suggested alt text:

SigNoz dashboard showing AI application latency and error rate

A useful dashboard should answer important questions quickly.

Avoid adding twenty charts only because they look impressive.

Four or five clear charts are better than a crowded dashboard that nobody understands.

Using Logs to Understand Errors

Metrics might tell you that ten requests failed.

A trace might show which operation failed.

Logs can show the exact message connected to that failure.

For example:

ERROR: AI provider request failed
Reason: Request timed out
Trace ID: abc123

This is where combining observability signals becomes powerful.

You can begin with a metric that shows an increased error rate, open a related trace, and then inspect the logs connected to the request.

Instead of searching through unrelated files, you can investigate the problem using connected information.

SigNoz supports collecting, searching, and analysing logs through OpenTelemetry.

Creating a Simple Alert

Developers cannot stare at dashboards all day.

Alerts tell you when something important changes.

A beginner alert could be:

Notify me when more than five percent of requests fail during a five-minute period.

Another example could be:

Notify me when the average response time becomes higher than three seconds.

SigNoz supports alerts based on telemetry data, including threshold and anomaly-based conditions.

Choose an alert that matches a real problem in your application.

Do not set the limit too low, or you may receive alerts for normal behaviour. Do not set it too high, or a serious problem may go unnoticed.

Add screenshot here: Your alert rule and its threshold.

Suggested alt text:

SigNoz alert for high AI chatbot response time

What I Learned

Before exploring SigNoz, observability sounded like something only large companies needed.

Now I understand why it matters even for small applications.

The three main signals work together:

• Logs tell me what happened.
• Metrics show the pattern over time.
• Traces show where one request spent its time.

The biggest lesson was that debugging becomes easier when I stop guessing.

Instead of reading random code and hoping to find the problem, I can begin with evidence from the running application.

I also learned that good observability does not mean collecting every possible piece of data.

It means collecting useful information that helps answer real questions.

Why SigNoz Is Interesting for AI Applications

AI applications can include many hidden steps:

• Model requests
• Prompt processing
• Tool calls
• Database searches
• Document retrieval
• API calls
• Response generation

The Agents of SigNoz hackathon focuses on making these workflows easier to observe. Its judging criteria reward strong use of SigNoz features such as traces, metrics, logs, dashboards, and alerts.

SigNoz is also built around OpenTelemetry.

That matters because OpenTelemetry is vendor-neutral. The instrumentation is not tied only to one monitoring company, making it easier to keep control over how telemetry is collected and where it is sent.

Frequently Asked Questions

What is SigNoz used for?

SigNoz is used to monitor applications and infrastructure. It combines traces, metrics, logs, exceptions, dashboards, and alerts in one observability platform.

Is SigNoz an AI model?

No.

SigNoz does not create chatbot answers or generate AI content. It monitors the AI application you build.

What is OpenTelemetry?

OpenTelemetry is an open-source framework for creating, collecting, and exporting telemetry such as traces, metrics, and logs.

What is the difference between SigNoz and OpenTelemetry?

OpenTelemetry collects and sends telemetry.

SigNoz receives, stores, searches, displays, and analyses that telemetry.

Can SigNoz monitor AI applications?

Yes. Developers can instrument AI applications, agents, model calls, and connected tools using OpenTelemetry and view the resulting traces, logs, and metrics in SigNoz. The hackathon specifically encourages AI-agent observability projects built around these capabilities.

What is distributed tracing?

Distributed tracing follows one request as it travels through different operations or services. It helps developers find where delays and errors happen.

Is SigNoz open source?

Yes. SigNoz describes itself as an open-source, OpenTelemetry-powered observability platform.

Final Thoughts

An AI application should not be a mystery box.

When something becomes slow, fails, or behaves unexpectedly, developers need to understand what happened inside the workflow.

OpenTelemetry collects that information.

SigNoz turns it into traces, metrics, logs, dashboards, and alerts that are easier to explore.

The feature I found most useful was distributed tracing because it changed one slow request from a mystery into a step-by-step story.

My next goal is to connect more parts of the AI workflow, including model calls, tool usage, errors, and token-related information, so that the complete application can be understood from one place.

Observability may sound complicated at first.