Monitoring Azure API Management metrics in a chatbot with Azure Functions

I use Azure API Management (AAM thereof) to provide my bot-builder APIs to chatbot developers. One of the important parts of it is to monitor usage, DAU (daily active users), most used endpoints, errors, and other metrics.

To do so AAM provides an Analytics tab where all information aggregated. The problem here that this is usage analytics, not product analytics, and it's also not very convenient to use.

AAM Analytics Chart

To solve this you could use the power of Application Insights to build a custom dashboard, or, as I prefer, built a simple reporting into a chatbot.

In this article, I describe my experience of how you could use Azure Functions to build product analytics and setup daily reports into a chatbot with it. We would connect and use Azure SDK with MSAL, HTTP-triggered and Timer-based functions, and API.chat proactive API.

Functions

You would need two Azure Functions with one ProductAnalytics service class. Service will be gathering and calculating analytics through Azure SDK and functions will provide data to the chatbot on request and on daily basis.

Service ProductAnalytics

To better understand AAM users you should monitor at least the next metrics

DAU - daily active users, how many users use your API yesterday
MAU - monthly active users, same but for month
Failed Request - how many requests failed, good to find some peaks.

You would most likely need a separate set of metrics for each AAM product you have.

That's how AnalyticsResult class might look

class AnalyticsResult
 {
     public int Dau { get; set; }
     public int Mau { get; set; }
     public int FailedRequests { get; set; }
 }

Credentials

Because you would use Azure SDK, first you should get credentials to access your AAM. To do so you should register a new application on the App registrations tab of your Azure Active Directory blade

Store Client ID, Tenant ID and create new Client secret on Certificates & secrets

Auth

To authenticate the client you could use several ways. The latest, and probably the most valid, way is to use MSAL.

Starting June 30th, 2020 we will no longer add any new features to Azure Active Directory Authentication Library (ADAL) and Azure AD Graph. We will continue to provide technical support and security updates but we will no longer provide feature updates. Applications will need to be upgraded to Microsoft Authentication Library (MSAL) and Microsoft Graph.

To use it, first, you need to add an Identity package

Install-Package Microsoft.Identity.Client -Version 4.28.0

And then you should implement a token provider for AAM to use in the Azure SDK client later, like this

public class ApiManagementTokenProvider : ITokenProvider
{
    private const string ScopeApiManagement = "https://management.azure.com/.default";

    private readonly string _tenantId;
    private readonly string _clientId;
    private readonly string _clientSecret;

    public ApiManagementTokenProvider()
    {
        // Bad example here, plz don't store secrets in code
        _tenantId = "...";
        _clientId = "...";
        _clientSecret = "...";
    }

    public async Task<AuthenticationHeaderValue> GetAuthenticationHeaderAsync(CancellationToken cancellationToken)
    {
        // For app only authentication, we need the specific tenant id in the authority url
        var tenantSpecificUrl = $"https://login.microsoftonline.com/{_tenantId}/";

        // Create a confidential client to authorize the app with the AAD app
        IConfidentialClientApplication clientApp = ConfidentialClientApplicationBuilder
                                                                        .Create(_clientId)
                                                                        .WithClientSecret(_clientSecret)
                                                                        .WithAuthority(tenantSpecificUrl)
                                                                        .Build();
        // Make a client call if Access token is not available in cache
        var authenticationResult = await clientApp
            .AcquireTokenForClient(new List<string> { ScopeApiManagement })
            .ExecuteAsync();

        return new AuthenticationHeaderValue("Bearer", authenticationResult.AccessToken);
    }
}

Client

To get Analytics from AAM with Azure SDK first you need to add relevant NuGet to your project

Install-Package Microsoft.Azure.Management.ApiManagement -Version 6.0.0-preview

To get the analytics you need to use the Reports endpoint.

These endpoints get data filtered by OData, so you would need to build several queries.

To get MAU and DAU you should use ListByUsers. Note what filter operation does it support - it's quite limited, to be honest.

Check this for example. Here is 100% valid code for creating ODataFilter, even better - it's built with LINQ and SetFilter method

var filter = new ODataQuery<ReportRecordContract>()
{
    OrderBy = "callCountTotal desc"
};
filter.SetFilter(i => i.ProductId == productId && i.Timestamp >= dateOnly && i.Timestamp <= dateEnd);

Here is a resulting filter string productId eq 'unlimited' and timestamp ge '2021-03-20T21:00:00Z' and timestamp le '2021-03-21T21:00:00Z'

But this filter string is not valid for Azure SDK! The error is Invalid filter clause specified: '...'.

It needs to be modified or rewritten like so to work: (timestamp ge 2021-03-21T21:00:00.000Z and timestamp le 2021-03-22T11:29:59.999Z) and (productId eq 'unlimited'). What the difference you ask? AAM OData doesn't like single quotes around dates.

Annoying thing: AAM OData require datetime in exact 2021-03-22T11:29:59.999Z format, which is neither standard UniversalSortableDateTimePattern or SortableDateTimePattern formats of DateTime.ToString().

Another annoying thing: ODataQuery does not allow to set $select parameter. Anyway,

DAU

var dateStart = date.Date;
var dateEnd = date.Date.AddDays(1);
var filter = new ODataQuery<ReportRecordContract>($"productId eq '{productId}' and timestamp ge {dateStart:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'} and timestamp le {dateEnd:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'}")
{
    OrderBy = "callCountTotal desc"
};

MAU

var dateStart = new DateTimeOffset(date.Year,date.Month,1,0,0,0,date.Offset);
var dateEnd = date.Date.AddDays(1);
var filter = new ODataQuery<ReportRecordContract>($"productId eq '{productId}' and timestamp ge {dateStart:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'} and timestamp le {dateEnd:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'}")
{
    OrderBy = "callCountTotal desc"
};

ListByUser call returns paginated data wrapped into the IPage interface. Page 'size' be changed by setting $skip and $top parameters. And I am not sure about the default max page size. That forces us to read page by page until we get a user with zero callCountTotal - because we need only active users. Full method

private async Task<List<ReportRecordContract>> GetReports(DateTimeOffset dateStart, DateTimeOffset dateEnd, string productId)
{
    var apiManagement = new ApiManagementClient(new TokenCredentials(_tokenProvider))
    {
        SubscriptionId = _subscriptionId
    };
    var filter = new ODataQuery<ReportRecordContract>($"productId eq '{productId}' and timestamp ge {dateStart:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'} and timestamp le {dateEnd:yyyy'-'MM'-'dd'T'HH':'mm':'ss'Z'}")
    {
        OrderBy = "callCountTotal desc",
    };

    var result = new List<ReportRecordContract>();
    bool hasMore = true;

    while (hasMore)
    {
        var page = await apiManagement.Reports.ListByUserAsync(filter, _resourceGroupName, _serviceName);
        result.AddRange(page);

        hasMore = page.Any() && page.Last().CallCountTotal > 0;
    }
    return result.ToList();
};

After you got our data you could calculate metrics.

Actual Functions

First of all, to make it easier you could override ToString of the AnalyticsResult class

        public override string ToString()
        {
            var sb = new StringBuilder();

            sb.AppendLine($"_DAU_: {Dau}");
            sb.AppendLine($"_MAU_: {Mau}");
            sb.AppendLine($"_Failed_: {FailedRequests}");

            return sb.ToString();
        }

Request

Let's start with a request. This function will receive a GET request and return a bot-compatible response. It will support today(default), yesterday, and date input which would be provided by the chatbot.

[FunctionName("GetDailyReports")]
public static async Task<IActionResult> Run(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "reports/daily")] HttpRequest req,
            ILogger log)
{
    log.LogInformation("C# HTTP trigger function processed a request.");

    DateTimeOffset date = DateTimeOffset.UtcNow.Date;
    string day = req.Query["day"];

    if (!string.IsNullOrEmpty(day) && !day.Equals("today", StringComparison.InvariantCultureIgnoreCase))
    {
        if (day.Equals("yesterday", StringComparison.InvariantCultureIgnoreCase))
            date = DateTimeOffset.UtcNow.AddDays(-1).Date;
        else
            DateTimeOffset.TryParse(day, out date);
    }

    var result = new StringBuilder();

    await AppendPlan(date, result, "*Personal Plan*", "personal");
    await AppendPlan(date, result, "*Team Plan*", "team");
    await AppendPlan(date, result, "*Business Plan*", "business");
    await AppendPlan(date, result, "*Ultimate Plan*", "ultimate");

    return new OkObjectResult(new { Messages = new List<string>() { result.ToString() } });

    static async Task AppendPlan(DateTimeOffset date, StringBuilder result, string title, string plan)
    {
        var service = new Services.AnalyticsService();
        result.AppendLine(title);
        var personalReport = await service.GetDailyAnalytics(date, plan);
        result.AppendLine(personalReport.ToString());
        result.AppendLine();
    }
}

Here you should first check the date parameter, then form an answer and send it to a user in a single message. You could split it into several messages, but IMO it would be a bit spammy.

Daily

The daily function is basically the same, but instead of responding to the chatbot, it sends a proactive message into the bot.

[FunctionName("SendDailyReport")]
public static async Task Run([TimerTrigger("0 0 13 * * ?")] TimerInfo myTimer, ILogger log)
{
    log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");

    var date = DateTimeOffset.UtcNow.Date;
    var result = new StringBuilder($"Report for *{date:d}*.");

    await AppendPlan(date, result, "*Personal Plan*", "personal");
    await AppendPlan(date, result, "*Team Plan*", "team");
    await AppendPlan(date, result, "*Business Plan*", "business");
    await AppendPlan(date, result, "*Unlimited Plan*", "unlimited");

    // send push to a chatbot
    await SendToChatbot(result.ToString());
}

For SendToChatbot you would need to obtain Subscription-Key, set botName and channel from the API.chat developer portal. The easiest way to do that is to navigate to Proactive API page, fill all data under your account, and copy generated C# code.

To obtain your chat id send a special /chatid command to the bot.

private static async Task SendToChatbot(string message)
{
    var client = new HttpClient();
    client.DefaultRequestHeaders.Add("Cache-Control", "no-cache");
    client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "{your-api-key}");

    var uri = "https://bot.api.chat/proactive/{your-bot}/{your-channel}/{your-chat-id}";

    using var content = new StringContent(JsonConvert.SerializeObject(new { Messages = new List<string>() { message } }), Encoding.UTF8, "application/json");
    var response = await client.PostAsync(uri, content);
}

Chatbot

For the chatbot, you could use my API.chat chatbot platform. The platform allows building a chatbot based on an XML-based scenario with the ability to perform third-party HTTP requests. Here is the simple scenario for the chatbot.

<bot>
  <state name="Start">
    <transition input="/report" next="Report" buttons="Today,Yesterday">Select or enter the day</transition>
    <transition input="*" next="Start" buttons="Report">echo: {msg}</transition>
  </state>
  <state name="Report">
    <transition input="*" morphology="msg" next="Start" action="http://contoso.azurewebsites.net/api/reports/daily?day={msg}"></transition>
  </state>
</bot>

User stays in Start state, send /report command and either press buttons or send a date. That will call the Azure function, provide answers and return the user into the Start state. Actually on any message in the Report state user will trigger the action. You could prevent it by expanding the scenario with regex validation of the user input.

<state name="Report">
    <transition input="today" next="Start" action="http://contoso.azurewebsites.net/api/reports/daily?day"/>
    <transition input="yesterday" next="Start" action="http://contoso.azurewebsites.net/api/reports/daily?day=yesterday"/>
    <transition input="@(0?[1-9]|[12][0-9]|3[01])[\/\-](0?[1-9]|1[012])[\/\-]\d{4}" morphology="msg" next="Start" action="http://contoso.azurewebsites.net/api/reports/daily?day={msg}"/>
    <transition input="*" morphology="msg" next="Report">Invalid date format.</transition>
</state>

All that is left is to connect API.chat chatbot to a telegram, and test it.

telegram demo

Future usage

From this point, the chatbot might be expanded even more. More complex analytics reports, more data, generated charts images - anything that you might need to monitor your Azure API Management usage.

One of the obvious improvements - rewrite GetDailyReports as an asynchronous HTTP Request, because building the report might take a lot of time if you have many users.

Monitoring Azure API Management metrics in a chatbot with Azure Functions

Comments

More from this blog

Building a Powerful MCP Server in .NET with Just 20 Lines of Code

WinUI 3: How to Set Minimum Window Size (Desktop)

Debugging .NET in Docker: Mount Points and Cross-platform Symbols

Harmonizing Line Endings Across Windows and WSL

Extending Visual Studio 2022

Functions

Service ProductAnalytics

Credentials

Auth

Client

Actual Functions

Request

Daily

Chatbot

Future usage

Command Palette

Comments

More from this blog

Functions

Service ProductAnalytics

Credentials

Auth

Client

Actual Functions

Request

Daily

Chatbot

Future usage