Tag Archives: Azure

Using TypeScript to write Cosmos DB stored procedures with async/await

Disclaimer: I am by no mean a TypeScript expert. In fact, I know very little about JS, npm, gulp, etc. So it’s entirely possible I said something really stupid in this article, or maybe I missed a much simpler way of doing things. Don’t hesitate to let me know in the comments!

Azure Cosmos DB (formerly known as Azure Document DB) is a NoSQL, multi-model, globally-distributed database hosted in Azure. If you come from relational SQL databases, it’s a very different world. Some things are great, for instance modeling data is much easier than in a relational database, and performance is excellent. Other things can be disconcerting, such as the lack of support for ACID. From the client’s perspective, there are no transactions: you can’t update multiple documents atomically. Of course, there’s a workaround: you can write stored procedures and triggers, which execute in the context of a transaction. So, when you really, really need multiple updates to be made atomically, you write a stored procedure to do the job.

The bad news

Unfortunately, on Cosmos DB, stored procedures are written… in Javascript 😢 (I know, plenty of folks love Javascript, but I don’t. Sue me!). All APIs for database operations are asynchronous (which is a good thing), but these APIs are based on callbacks, not on promises, so even though ECMAScript 2017 is supported, you can’t use async/await with them. This fact is enough to turn any non-trivial task (i.e. code that involves branches such as ifs or loops) into a nightmare, at least for a C# developer like me… I typically spend a full day to write and debug a stored procedure that should have taken less than an hour with async/await.

Promise-based wrapper

Of course, I wouldn’t be writing this post if there wasn’t a way to make things better. Cosmos DB Product Manager Andrew Liu was kind enough to show me how to write a wrapper around the callback-based API to enable the use of promises and async/await. Basically, it’s just a few functions that you can add to your stored procedures:

function setFoo() {
    async function main() {
        let { feed, options } = await queryDocuments("SELECT * from c");
        for (let doc of feed) {
            doc.foo = "bar";
            await replaceDocument(doc);
        }
    }

    main().catch(err => getContext().abort(err));
}

function queryDocuments(sqlQuery, options) {
    return new Promise((resolve, reject) => {
        let isAccepted = __.queryDocuments(__.getSelfLink(), sqlQuery, options, (err, feed, opts) => {
            if (err) reject(err);
            else resolve({ feed, options: opts });
        });
        if (!isAccepted) reject(new Error(429, "queryDocuments was not accepted."));
    });
}

function replaceDocument(doc, options) {
    return new Promise((resolve, reject) => {
        let isAccepted = __.replaceDocument(doc._self, doc, (err, result, opts) => {
            if (err) reject(err);
            else resolve({ result, options: opts });
        });
        if (!isAccepted) reject(new Error(429, "replaceDocument was not accepted."));
    });
}

// and so on for other APIs...

Note that the stored procedure’s entry point (setFoo in this example) cannot be async (if it returns a promise, Cosmos DB won’t wait for it to complete), so you need to write another async function (main), call it from the stored procedure’s entry point, and catch the error that could be thrown. Note the use of getContext().abort(err), which aborts and rolls back the current transaction; without this, the exception would be swallowed.

I’m not going to show the equivalent code using the callback-based API here, because honestly, it makes my head hurt just thinking about it. But trust me on this: it’s not pretty, and much harder to understand.

Using TypeScript

The code shown above is pretty straightforward, once you have the wrapper functions. However, there are at least two issues with it:

  • This is still Javascript, which is weakly typed, so it’s easy to make mistakes that won’t be caught until runtime.
  • Cosmos DB stored procedures and triggers must consist of a single self-contained file; no import or require allowed. Which means you can’t share the wrapper functions across multiple stored procedures, you have to include them in each stored procedure. This is annoying…

First, let’s see how we can write our stored procedure in TypeScript and reduce the boilerplate code.

Let’s start by installing TypeScript. Create a package.json file with the npm init command (it will prompt you for a few details, you can leave everything empty), and run the npm install typescript command. We’ll also need the TypeScript definitions of the Cosmos DB server-side APIs. For this, we’ll install a npm package named @types/documentdb-server which contains the definitions: npm install @types/documentdb-server.

We also need a tsconfig.json file:

{
    "exclude": [
        "node_modules"
    ],
    "compilerOptions": {
        "target": "es2017",
        "strict": true,
    }
}

Now, let’s create a few helpers to use in our stored procedures. I put them all in a CosmosServerScriptHelpers folder. The most important piece is the AsyncCosmosContext class, which is basically a strongly-typed, promise-based wrapper for the __ object. It implements the following interface:

export interface IAsyncCosmosContext {

    readonly request: IRequest;
    readonly response: IResponse;

    // Basic query and CRUD methods
    queryDocuments(sqlQuery: any, options?: IFeedOptions): Promise<IFeedResult>;
    readDocument(link: string, options?: IReadOptions): Promise<any>;
    createDocument(doc: any, options?: ICreateOptions): Promise<any>;
    replaceDocument(doc: any, options?: IReplaceOptions): Promise<any>;
    deleteDocument(doc: any, options?: IDeleteOptions): Promise<any>;

    // Helper methods
    readDocumentById(id: string, options?: IReadOptions): Promise<any>;
    readDocumentByIdIfExists(id: string, options?: IReadOptions): Promise<any>;
    deleteDocumentById(id: string, options?: IDeleteOptions): Promise<any>
    queryFirstDocument(sqlQuery: any, options?: IFeedOptions): Promise<any>;
    createOrReplaceDocument(doc: any, options?: ICreateOrReplaceOptions): Promise<any>;
}

I’m not showing the whole code in this article because it would be too long, but you can see the implementation and auxiliary types in the GitHub repo here: https://github.com/thomaslevesque/TypeScriptCosmosDBStoredProceduresArticle.

So, how can we use this? Let’s look at our previous example again, and see how we can rewrite it in TypeScript using our wrappers:

import {IAsyncCosmosContext} from "CosmosServerScriptHelpers/IAsyncCosmosContext";
import {AsyncCosmosContext} from "CosmosServerScriptHelpers/AsyncCosmosContext";

function setFoo() {
    async function main(context: IAsyncCosmosContext) {
        let { feed, options } = await context.queryDocuments("SELECT * from c");
        for (let doc of feed) {
            doc.foo = "bar";
            await replaceDocument(doc);
        }
    }

    main(new AsyncCosmosContext()).catch(err => getContext().abort(err));
}

It looks remarkably similar to the previous version, with just the following changes:

  • We no longer have the wrapper functions in the same file, instead we just import them via the AsyncCosmosContext class.
  • We pass an instance of AsyncCosmosContext to the main function.

This looks pretty good already, but what’s bugging me is having to explicitly create the context and do the .catch(...). So let’s create another helper to encapsulate this:

import {IAsyncCosmosContext} from "./IAsyncCosmosContext";
import {AsyncCosmosContext} from "./AsyncCosmosContext";

export class AsyncHelper {
    /**
     * Executes the specified async function and returns its result as the response body of the stored procedure.
     * @param func The async function to execute, which returns an object.
     */
    public static executeAndReturn(func: (context: IAsyncCosmosContext) => Promise<any>) {
        this.executeCore(func, true);
    }

    /**
     * Executes the specified async function, but doesn't write anything to the response body of the stored procedure.
     * @param func The async function to execute, which returns nothing.
     */
    public static execute(func: (context: IAsyncCosmosContext) => Promise<void>) {
        this.executeCore(func, false);
    }

    private static executeCore(func: (context: IAsyncCosmosContext) => Promise<any>, setBody: boolean) {
        func(new AsyncCosmosContext())
            .then(result => {
                if (setBody) {
                    __.response.setBody(result);
                }
            })
            .catch(err => {
                // @ts-ignore
                getContext().abort(err);
            });
    }
}

Using this helper, our stored procedure now looks like this:

import {AsyncHelper} from "CosmosServerScriptHelpers/AsyncHelper";

function setFoo() 
{
    AsyncHelper.execute(async context => {
        let result = await context.queryDocuments("SELECT * from c");
        for (let doc of result.feed) {
            doc.foo = "bar";
            await context.replaceDocument(doc);
        }
    });
}

This reduces the boilerplate code to a minimum. I’m pretty happy with it, so let’s leave it alone.

Generate the actual JS stored procedure files

OK, now comes the tricky part… We have a bunch of TypeScript files that import each other. But Cosmos DB wants a single, self-contained JavaScript file, with the first function as the entry point of the stored procedure. By default, compiling the TypeScript files to JavaScript will just generate one JS file for each TS file. The --outFile compiler option outputs everything to a single file, but it doesn’t really work for us, because it still emits some module related code that won’t work in Cosmos DB. What we need, for each stored procedure, is a file that only contains:

  • the stored procedure function itself
  • all the helper code, without any import or require.

Since it doesn’t seem possible to get the desired result using just the TypeScript compiler, the solution I found was to use a Gulp pipeline to concatenate the output files and remove the extraneous exports and imports. Here’s my gulpfile.js:

const gulp = require("gulp");
const ts = require("gulp-typescript");
const path = require("path");
const flatmap = require("gulp-flatmap");
const replace = require('gulp-replace');
const concat = require('gulp-concat');

gulp.task("build-cosmos-server-scripts", function() {
    const sharedScripts = "CosmosServerScriptHelpers/*.ts";
    const tsServerSideScripts = "StoredProcedures/**/*.ts";

    return gulp.src(tsServerSideScripts)
        .pipe(flatmap((stream, file) =>
        {
            let outFile = path.join(path.dirname(file.relative), path.basename(file.relative, ".ts") + ".js");
            let tsProject = ts.createProject("tsconfig.json");
            return stream
                .pipe(gulp.src(sharedScripts))
                .pipe(tsProject())
                .pipe(replace(/^\s*import .+;\s*$/gm, ""))
                .pipe(replace(/^\s*export .+;\s*$/gm, ""))
                .pipe(replace(/^\s*export /gm, ""))
                .pipe(concat(outFile))
                .pipe(gulp.dest("StoredProcedures"));
        }));
});

gulp.task("default", gulp.series("build-cosmos-server-scripts"));

Note that this script requires a few additional npm packages: gulp, gulp-concat, gulp-replace, gulp-flatmap, and gulp-typescript.

Now you can just run gulp and it will produce the appropriate JS file for each TS stored procedure.

To be honest, this solution feels a bit hacky, but it’s the best I’ve been able to come up with. If you know of a better approach, please let me know!

Wrapping up

The out-of-the-box experience for writing Cosmos DB server-side code is not great (to put it mildly), but with just a bit of work, it can be made much better. You can have strong-typing thanks to TypeScript and the type definitions, and you can use async/await to make the code simpler. Note that this approach is also valid for triggers.

Hopefully, a future Cosmos DB update will introduce a proper promise-based API, and maybe even TypeScript support. In the meantime, feel free to use the solution in this post!

The full code for this article is here: https://github.com/thomaslevesque/TypeScriptCosmosDBStoredProceduresArticle.

Scaling out ASP.NET Core SignalR using Azure Service Bus

ASP.NET Core SignalR is a super easy way to establish two-way communication between an ASP.NET Core app and its clients, using WebSockets, Server-Sent Events, or long polling, depending on the client’s capabilities. For instance, it can be used to send a notification to all connected clients. However, if you scale out your application to multiple server instances, it no longer works out of the box: only the clients connected to the instance that sent the notification will receive it. Microsoft has two documented solutions to this problem:

Derek Comartin did a good job explaining these solutions (Redis, Azure SignalR Service), so I won’t go into the details. Both are perfectly viable, however they’re relatively expensive. A Redis Cache resource in Azure starts at about 14€/month for the smallest size, and Azure SignalR Service starts at about 40€/month for a single unit (I’m entirely dismissing the free plan, which is too limited to use beyond development scenarios). Sure, it’s not that expensive, but why pay more when you can pay less?

What I want to talk about in this post is a third option that will probably be cheaper in many scenarios: using Azure Service Bus to dispatch SignalR messages between server instances. In fact, this approach was supported in classic ASP.NET, but it hasn’t been ported to ASP.NET Core.

Here’s an overview of how one could manually implement the Azure Service Bus approach:

  • When an instance of the application wants to send a SignalR message to all clients, it sends it:

    • via its own SignalR hub or hub context (only clients connected to this instance will receive it)
    • and to an Azure Service Bus topic, for distribution to other instances.
    // Pseudo code...
    
    private readonly IHubContext<ChatHub, IChatClient> _hubContext;
    private readonly IServiceBusPublisher _serviceBusPublisher;
    
    public async Task SendMessageToAllAsync(string text)
    {
        // Send the message to clients connected to the current instance
        await _hubContext.Clients.All.ReceiveMessageAsync(text);
    
        // Notify other instances to send the same message
        await _serviceBusPublisher.PublishMessageAsync(new SendToAllMessage(text));
    }
    
  • Each instance of the application runs a hosted service that subscribes to the topic and processes the messages

    • When a message is received, it’s sent to the relevant clients via the hub context, unless it’s from the current instance.
    // Very simplified pseudo code...
    
    // Subscribe to the topic
    var subscriptionClient = new SubscriptionClient(connectionString, topicName, subscriptionName);
    subscriptionClient.RegisterMessageHandler(OnMessageReceived, OnError);
    
    ...
    
    private async Task OnMessageReceived(Message sbMessage, CancellationToken cancellationToken)
    {
        SendToAllMessage message = DeserializeServiceBusMessage(sbMessage);
    
        if (message.SenderInstanceId == MyInstanceId)
            return; // ignore message from self
    
        // Send the message to clients connected to the current instance
        await _hubContext.Clients.All.ReceiveMessageAsync(message.Text);
    }
    

I’m not showing the full details of how to implement this solution, because to be honest, it kind of sucks. It works, but it’s a bit ugly: the fact that it’s using a service bus to share messages with other server instances is too visible, you can’t just ignore it. Every time you send a message via SignalR, you also have to explicitly send one to the service bus. It would be better to hide that ugliness behind an abstraction, or even better, make it completely invisible…

If you have used the Redis or Azure SignalR Service approaches before, you might have noticed how simple they are to use. Basically, in your Startup.ConfigureServices method, just append AddRedis(...) or AddAzureSignalR(...) after services.AddSignalR(), and you’re done: you can use SignalR as usual, the details of how it handles scale-out are completely abstracted away. Wouldn’t it be nice to be able to do the same for Azure Service Bus? I thought so too, so I made a library that does exactly that: AspNetCore.SignalR.AzureServiceBus. To use it, reference the NuGet package, and just add this in your Startup.ConfigureServices method:

services.AddSignalR()
        .AddAzureServiceBus(options =>
        {
            options.ConnectionString = "(your service bus connection string)";
            options.TopicName = "(your topic name)";
        });

Disclaimer: The library is still in alpha status, probably not ready for production use. I’m not aware of any issue, but it hasn’t been battle tested yet. Use at your own risk, and please report any issues you find!

Multitenant Azure AD issuer validation in ASP.NET Core

If you use Azure AD authentication and want to allow users from any tenant to connect to your ASP.NET Core application, you need to configure the Azure AD app as multi-tenant, and use a "wildcard" tenant id such as organizations or common in the authority URL:

openIdConnectOptions.Authority = "https://login.microsoftonline.com/organizations/v2.0";

The problem when you do that is that with the default configuration, the token validation will fail because the issuer in the token won’t match the issuer specified in the OpenID metadata. This is because the issuer from the metadata includes a placeholder for the tenant id:

https://login.microsoftonline.com/{tenantid}/v2.0

But the iss claim in the token contains the URL for the actual tenant, e.g.:

https://login.microsoftonline.com/64c5f641-7e94-4d21-ae5c-9747994e4211/v2.0

A workaround that is often suggested is to disable issuer validation in the token validation parameters:

openIdConnectOptions.TokenValidationParameters.ValidateIssuer = false;

However, if you do that the issuer won’t be validated at all. Admittedly, it’s not much of a problem, since the token signature will prove the issuer identity anyway, but it still bothers me…

Fortunately, you can control how the issuer is validated, by specifying the TokenValidator property:

options.TokenValidationParameters.IssuerValidator = ValidateIssuerWithPlaceholder;

Where ValidateIssuerWithPlaceholder is the method that validates the issuer. In that method, we need to check if the issuer from the token matches the issuer with a placeholder from the metadata. To do this, we just replace the {tenantid} placeholder with the value of the token’s tid claim (which contains the tenant id), and check that the result matches the token’s issuer:

private static string ValidateIssuerWithPlaceholder(string issuer, SecurityToken token, TokenValidationParameters parameters)
{
    // Accepts any issuer of the form "https://login.microsoftonline.com/{tenantid}/v2.0",
    // where tenantid is the tid from the token.

    if (token is JwtSecurityToken jwt)
    {
        if (jwt.Payload.TryGetValue("tid", out var value) &&
            value is string tokenTenantId)
        {
            var validIssuers = (parameters.ValidIssuers ?? Enumerable.Empty<string>())
                .Append(parameters.ValidIssuer)
                .Where(i => !string.IsNullOrEmpty(i));

            if (validIssuers.Any(i => i.Replace("{tenantid}", tokenTenantId) == issuer))
                return issuer;
        }
    }

    // Recreate the exception that is thrown by default
    // when issuer validation fails
    var validIssuer = parameters.ValidIssuer ?? "null";
    var validIssuers = parameters.ValidIssuers == null
        ? "null"
        : !parameters.ValidIssuers.Any()
            ? "empty"
            : string.Join(", ", parameters.ValidIssuers);
    string errorMessage = FormattableString.Invariant(
        $"IDX10205: Issuer validation failed. Issuer: '{issuer}'. Did not match: validationParameters.ValidIssuer: '{validIssuer}' or validationParameters.ValidIssuers: '{validIssuers}'.");

    throw new SecurityTokenInvalidIssuerException(errorMessage)
    {
        InvalidIssuer = issuer
    };
}

With this in place, you’re now able to fully validate tokens from any Azure AD tenant without skipping issuer validation.

Happy coding, and merry Christmas!

Writing a GitHub Webhook as an Azure Function

I recently experimented with Azure Functions and GitHub apps, and I wanted to share what I learned.

A bit of background

As you may already know, I’m one of the maintainers of the FakeItEasy mocking library. As is common in open-source projects, we use a workflow based on feature branches and pull requests. When a change is requested in a PR during code review, we usually make the change as a fixup commit, because it makes it easier to review, and because we like to keep a clean history. When the changes are approved, the author squashes the fixup commits before the PR is merged. Unfortunately, I’m a little absent minded, and when I review a PR, I often forget to wait for the author to squash their commits before I merge… This causes the fixup commits to appear in the main dev branch, which is ugly.

Which leads me to the point of this post: I wanted to make a bot that could prevent a PR from being merged if it had commits that needed to be squashed (i.e. commits whose messages start with fixup! or squash!). And while I was at it, I thought I might as well make it usable by everyone, so I made it a GitHub app: DontMergeMeYet.

GitHub apps

Now, you might be wondering, what on Earth is a GitHub app? It’s simply a third-party application that is granted access to a GitHub repository using its own identity; what it can do with the repo depends on which permissions were granted. A GitHub app can also receive webhook notifications when events occur in the repo (e.g. a comment is posted, a pull request is opened, etc.).

A GitHub app could, for instance, react when a pull request is opened or updated, examine the PR details, and add a commit status to indicate whether the PR is ready to merge or not (this WIP app does this, but doesn’t take fixup commits into account).

As you can see, it’s a pretty good fit for what I’m trying to do!

In order to create a GitHub app, you need to go to the GitHub apps page, and click New GitHub app. You then fill in at least the name, homepage, and webhook URL, give the app the necessary permissions, and subscribe to the webhook events you need. In my case, I only needed read-only access to pull requests, read-write access to commit statuses, and to receive pull request events.

At this point, we don’t yet have an URL for the webhook, so enter any valid URL; we’ll change it later after we actually implemented the app.

Azure Functions

I hadn’t paid much attention to Azure Functions before, because I didn’t really see the point. So I started to implement my webhook as a full-blown ASP.NET Core app, but then I realized several things:

  • My app only had a single HTTP endpoint
  • It was fully stateless and didn’t need a database
  • If I wanted the webhook to always respond quickly, the Azure App Service had to be "always on"; that option isn’t available in free plans, and I didn’t want to pay a fortune for a better service plan.

I looked around and realized that Azure Functions had a "consumption plan", with a generous amount (1 million per month) of free requests before I had to pay anything, and functions using this plan are "always on". Since I had a single endpoint and no persistent state, an Azure Function seemed to be the best fit for my requirements.

Interestingly, Azure Functions can be triggered, among other things, by GitHub webhooks. This is very convenient as it takes care of validating the payload signature.

So, Azure Functions turn out to be a perfect match for implementing my webhook. Let’s look at how to create one.

Creating an Azure Function triggered by a GitHub webhook

It’s possible to write Azure functions in JavaScript, C# (csx) or F# directly in the portal, but I wanted the comfort of the IDE, so I used Visual Studio. To write an Azure Function in VS, follow the instructions on this page. When you create the project, a dialog appears to let you choose some options:

New function dialog

  • version of the Azure Functions runtime: v1 targets the full .NET Framework, v2 targets .NET Core. I picked v1, because I had trouble with the dependencies in .NET Core.
  • Trigger: GitHub webhooks don’t appear here, so just pick "HTTP Trigger", we’ll make the necessary changes in the code.
  • Storage account: pick the storage emulator; when you publish the function, a real Azure storage account will be set instead
  • Access rights: it doesn’t matter what you pick, we’ll override it in the code.

The project template creates a class named Function1 with a Run method that looks like this:

public static class Function1
{
    [FunctionName("Function1")]
    public static async Task<HttpResponseMessage> Run(
        [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
    {
        ...
    }
}

Rename the class to something that makes more sense, e.g. GitHubWebHook, and don’t forget to change the name in the FunctionName attribute as well.

Now we need to tell the Azure Functions runtime that this function is triggered by a GitHub webhook. To do this, change the method signature to look like this:

    [FunctionName("GitHubWebHook")]
    public static async Task<HttpResponseMessage> Run(
        [HttpTrigger("POST", WebHookType = "github")] HttpRequestMessage req,
        TraceWriter log)

GitHub webhooks always use the HTTP POST method; the WebHookType property is set to "github" to indicate that it’s a GitHub webhook.

Note that it doesn’t really matter what we respond to the webhook request; GitHub doesn’t do anything with the response. I chose to return a 204 (No content) response, but you can return a 200 or anything else, it doesn’t matter.

Publishing the Azure Function

To publish your function, just right click on the Function App project, and click Publish. This will show a wizard that will let you create a new Function App resource on your Azure subscription, or select an existing one. Not much to explain here, it’s pretty straightforward; just follow the wizard!

When the function is published, you need to tell GitHub how to invoke it. Open the Azure portal in your browser, navigate to your new Function App, and select the GitHubWebHook function. This will show the content of the (generated) function.json file. Above the code view, you will see two links: Get function URL, and Get GitHub secret:

Azure Function URL and secret

You need to copy the URL to the Webhook URL field in the GitHub app settings, and copy the secret to the Webhook secret field. This secret is used to calculate a signature for webhook payloads, so that the Azure Function can ensure the payloads really come from GitHub. As I mentioned earlier, this verification is done automatically when you use a GitHub HTTP trigger.

And that’s it, your webhook is online! Now you can go install the GitHub app into one of your repositories, and your webhook will start receiving events for this repo.

Points of interest

I won’t describe the whole implementation of my webhook in this post, because it would be too long and most of it isn’t that interesting, but I will just highlight a few points of interest. You can find the complete code on GitHub.

Parsing the payload

Rather than reinventing the wheel, we can leverage the Octokit .NET library. Octokit is a library made by GitHub to consume the GitHub REST API. It contains classes representing the entities used in the API, including webhook payloads, so we can just deserialize the request content as a PullRequestEventPayload. However, if we just try to do this with JSON.NET, this isn’t going to work: Octokit doesn’t use JSON.NET, so the classes aren’t decorated with JSON.NET attributes to map the C# property names to the JSON property names. Instead, we need to use the JSON serializer that is included in Octokit, called SimpleJsonSerializer:

private static async Task<PullRequestEventPayload> DeserializePayloadAsync(HttpContent content)
{
    string json = await content.ReadAsStringAsync();
    var serializer = new SimpleJsonSerializer();
    return serializer.Deserialize<PullRequestEventPayload>(json);
}

There’s also another issue: the PullRequestEventPayload from Octokit is missing the Installation property, which we’re going to need later to authenticate with the GitHub API. An easy workaround is to make a new class that inherits PullRequestEventPayload and add the new property:

public class PullRequestPayload : PullRequestEventPayload
{
    public Installation Installation { get; set; }
}

public class Installation
{
    public int Id { get; set; }
}

And we’ll just use PullRequestPayload instead of PullRequestEventPayload.

Authenticating with the GitHub API

We’re going to need to call the GitHub REST API for two things:

  • to get the list of commits in the pull request
  • to update the commit status

In order to access the API, we’re going to need credentials… but which credentials? We could just generate a personal access token and use that, but then we would access the API as a "real" GitHub user, and we would only be able to access our own repositories (for writing, at least).

As I mentioned earlier, GitHub apps have their own identity. What I didn’t say is that when authenticated as themselves, there isn’t much they’re allowed to do: they can only get management information about themselves, and get a token to authenticate as an installation. An installation is, roughly, an instance of the application that is installed on one or more repo. When someone installs your app on their repo, it creates an installation. Once you get a token for an installation, you can access all the APIs allowed by the app’s permissions on the repos it’s installed on.

However, there are a few hoops to jump through to get this token… This page describes the process in detail.

The first step is to generate a JSON Web Token (JWT) for the app. This token has to contain the following claims:

  • iat: the timestamp at which the token was issued
  • exp: the timestamp at which the token expires
  • iss: the issuer, which is actually the app ID (found in the GitHub app settings page)

This JWT needs to be signed with the RS256 algorithm (RSA signature with SHA256); in order to sign it, you need a private key, which must be generated from the GitHub app settings page. You can download the private key in PEM format, and store it somewhere your app can access it. Unfortunately, the .NET APIs to generate and sign a JWT don’t handle the PEM format, they need an RSAParameters object… But Stackoverflow is our friend, and this answer contains the code we need to convert a PEM private key to an RSAParameters object. I just kept the part I needed, and manually reformatted the PEM private key to remove the header, footer, and newlines, so that it could easily be stored in the settings as a single line of text.

Once you have the private key as an RSAParameters object, you can generate a JWT like this:

public string GetTokenForApplication()
{
    var key = new RsaSecurityKey(_settings.RsaParameters);
    var creds = new SigningCredentials(key, SecurityAlgorithms.RsaSha256);
    var now = DateTime.UtcNow;
    var token = new JwtSecurityToken(claims: new[]
        {
            new Claim("iat", now.ToUnixTimeStamp().ToString(), ClaimValueTypes.Integer),
            new Claim("exp", now.AddMinutes(10).ToUnixTimeStamp().ToString(), ClaimValueTypes.Integer),
            new Claim("iss", _settings.AppId)
        },
        signingCredentials: creds);

    var jwt = new JwtSecurityTokenHandler().WriteToken(token);
    return jwt;
}

A few notes about this code:

  • It requires the following NuGet packages:
    • Microsoft.IdentityModel.Tokens 5.2.1
    • System.IdentityModel.Tokens.Jwt 5.2.1
  • ToUnixTimeStamp is an extension method that converts a DateTime to a UNIX timestamp; you can find it here
  • As per the GitHub documentation, the token lifetime cannot exceed 10 minutes

Once you have the JWT, you can get an installation access token by calling the "new installation token" API endpoint. You can authenticate to this endpoint by using the generated JWT as a Bearer token

public async Task<string> GetTokenForInstallationAsync(int installationId)
{
    var appToken = GetTokenForApplication();
    using (var client = new HttpClient())
    {
        string url = $"https://api.github.com/installations/{installationId}/access_tokens";
        var request = new HttpRequestMessage(HttpMethod.Post, url)
        {
            Headers =
            {
                Authorization = new AuthenticationHeaderValue("Bearer", appToken),
                UserAgent =
                {
                    ProductInfoHeaderValue.Parse("DontMergeMeYet"),
                },
                Accept =
                {
                    MediaTypeWithQualityHeaderValue.Parse("application/vnd.github.machine-man-preview+json")
                }
            }
        };
        using (var response = await client.SendAsync(request))
        {
            response.EnsureSuccessStatusCode();
            var json = await response.Content.ReadAsStringAsync();
            var obj = JObject.Parse(json);
            return obj["token"]?.Value<string>();
        }
    }
}

OK, almost there. Now we just need to use the installation token to call the GitHub API. This can be done easily with Octokit:

private IGitHubClient CreateGitHubClient(string installationToken)
{
    var userAgent = new ProductHeaderValue("DontMergeMeYet");
    return new GitHubClient(userAgent)
    {
        Credentials = new Credentials(installationToken)
    };
}

And that’s it, you can now call the GitHub API as an installation of your app.

Note: the code above isn’t exactly what you’ll find in the repo; I simplified it a little for the sake of clarity.

Testing locally using ngrok

When creating your Azure Function, it’s useful to be able to debug on your local machine. However, how will GitHub be able to call your function if it doesn’t have a publicly accessible URL? The answer is a tool called ngrok. Ngrok can create a temporary host name that forwards all traffic to a port on your local machine. To use it, create an account (it’s free) and download the command line tool. Once logged in to the ngrok website, a page will give you the command to save an authentication token on your machine. Just execute this command:

ngrok authtoken 1beErG2VTJJ0azL3r2SBn_2iz8johqNv612vaXa3Rkm

Start your Azure Function in debug from Visual Studio; the console will show you the local URL of the function, something like http://localhost:7071/api/GitHubWebHook. Note the port, and in a new console, start ngrok like this:

ngrok http 7071 --host-header rewrite

This will create a new hostname and start forwarding traffic to the 7071 port on your machine. The --host-header rewrite argument causes ngrok to change the Host HTTP header to localhost, rather than the temporary hostname; Azure Functions don’t work correctly without this.

You can see the temporary hostname in the command output:

ngrok by @inconshreveable                                                                                                                                                                                         (Ctrl+C to quit)

Session Status                online
Account                       Thomas Levesque (Plan: Free)
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://89e14c16.ngrok.io -> localhost:7071
Forwarding                    https://89e14c16.ngrok.io -> localhost:7071

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Finally, go to the GitHub app settings, and change the webook URL to https://89e14c16.ngrok.io/api/GitHubWebHook (i.e. the temporary domain with the same path as the local URL).

Now you’re all set. GitHub will send the webhook payloads to ngrok, which will forward them to your app running locally.

Note that unless you have a paid plan for ngrok, the temporary subdomain changes every time you start the tool, which is annoying. So it’s better to keep it running for the whole development session, otherwise you will need to change the GitHub app settings again.

Conclusion

Hopefully you learned a few things from this article. With Azure Functions, it’s almost trivial to implement a GitHub webhook (the only tricky part is the authentication to call the GitHub API, but not all webhooks need it). It’s much lighter than a full-blown web app, and much simpler to write: you don’t have to care about MVC, routing, services, etc. And if it wasn’t enough, the pricing model for Azure Functions make it a very cheap option for hosting a webhook!