Infrastructure with DotNet Core 3.0

Using Pulumi to automate home automation recipes

In a previous two-part blog series I wrote about building a home automation recipe and automating its deployment with a Pulumi app. That was written in TypeScript. Now that Pulumi supports DotNet Core 3.0 with Pulumi CLI 1.5.0+, I ported over the infrastructure to C# as well, and added it as a C# project to my existing Solution.

A C# app and a C# infrastructure app. Nice. It was worth the wait.

What do you think would have happened when I ran pulumi up in the C# version of the same infrastructure? You wouldn’t expect any changes since the infrastructure wasn’t changed — just the language that it was written in, right? Well, that’s exactly what happened (or didn’t happen, depending on how you look at it.) Nothing changed when I ran a pulumi preview after I completed porting over the application.

That, my friends, is the power of programming languages…and proper infrastructure as code.

Building Home Automation Recipes with Pulumi

Using Pulumi to automate home automation recipes

Home automation is easier than ever with a plethora of IoT-connected devices enabling everyday appliances in your home to be internet-enabled. When you write a custom piece of integration for your IoT, often times deployment to the cloud becomes an afterthought, only to become a nightmare when you actually want to update it later, and you don’t remember where/how you deployed it. With Pulumi, you don’t have to worry about that anymore. You can develop your IoT integration app, as well as your program your infrastructure as an app.

The Garage Door Opener

Most people have an internet hub connected to their automatic garage door opener, which allows them to remotely monitor the garage door as well as open/close it using a mobile app. But what about when you forget to close it and it stays open? Neither the app nor the existing home automation recipes on the home automation website IFTTT have a way to remind you that you left it open. To solve this problem I tried not to build something of my own, but instead try to use Zapier, a task automation platform.

Note: The source code for this post is available here.

The First Attempt

My first attempt involved using Zapier. It would have worked if there was a way to update a state while waiting for a timer to fire. I used IFTTT to connect the myQ service to fire a Webhook request each time the garage door opened or closed. The webhook receiver was a Zapier webhook “catch” action, which I then connected to a timer delay before sending me a text message via Twilio. It mostly worked, except, if I closed the garage door before the timer fires, there was no way for me to update the state and therefore, cancel sending the text message.

Here’s the “zap” I ended up creating:

A Zap on Zapier using built-in actions.

Durable Functions on Azure Functions

Durable Functions is an extension to the already popular Azure Functions platform. This means, you can write functions with an external trigger (HTTP, Queue etc.) and have it trigger an orchestration. Each orchestration instance is automatically tracked by the platform. Checkout the API reference to see what you can control about an orchestration instance.

Durable Function Types

There are other durable function types, learn more about them here. The following are just the types used in this project.

Orchestration Functions

Each function has a trigger type that identifies _how_ that function can be triggered. Orchestration functions are no different. Orchestration functions typically don’t do any work other than, you guessed it, orchestrate other functions that do the work.

Activity Functions

Activity functions are responsible for most of the work in an orchestration. You can make HTTP calls, call other activity functions etc.

Entity Functions

Entity functions are only available as part of the Durable Functions 2.x beta. This is in public preview.

Entity functions allow you to represent your orchestration instance with a state. It is up to you on whether each orchestration instance has its own entity or if your state is a singleton. This is controlled by the way that entities are persisted. Each entity is made up of two components:

  • An entity name: a name that identifies the type of the entity (for example, “Counter”).
  • An entity key: a string that uniquely identifies the entity among all other entities of the same name (for example, a GUID).

IFTTT + Azure Durable Functions + Twilio + Pulumi

This is the high-level view of the solution I finally ended up with.

  • IFTTT receives signals from the garage door opener.
  • IFTTT then calls the function app.
  • The function app waits for couple of minutes and if the door still isn’t closed by then, sends a text message using Twilio.

Function App

The only external trigger in the function app is the HTTP trigger used in this function. An orchestration instance is created only through the orchestration client, injected into the HTTP-triggered function as a param.

[FunctionName("Main")]
public static async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext ctx,
    ILogger log)
{
    var delay = Environment.GetEnvironmentVariable("TimerDelayMinutes");
    var delayTimeSpan = TimeSpan.FromMinutes(Convert.ToInt32(delay));
    DateTime timer = ctx.CurrentUtcDateTime.Add(delayTimeSpan);
    log.LogInformation($"Setting timer to expire at {timer.ToLocalTime().ToString()}");
    await ctx.CreateTimer(timer, CancellationToken.None);

    try
    {
        // The use of a critical block, though optional, is recommended here.
        // Updates to durable entities are serial, by default.
        // Having the lock ensures that the entity state we are reading is guaranteed to
        // be the current value of the entity.
        using (await ctx.LockAsync(EntityId))
        {
            var currentState = await ctx.CallEntityAsync<string>(EntityId, "read", null);
            log.LogInformation($"Current state is {currentState}.");
            // If the door is closed already, then don't do anything.
            if (currentState.ToLowerInvariant() == "closed")
            {
                log.LogInformation("Looks like the door was already closed. Will skip sending text message.");
                return;
            }
            await ctx.CallActivityAsync("SendTextMessage", null);
        }
    }
    catch (LockingRulesViolationException ex)
    {
        log.LogError(ex, "Failed to lock/call the entity.");
    }
    catch (Exception ex)
    {
        log.LogError(ex, "Unexpected exception occurred.");
    }
}

Deploying the Infrastructure using Pulumi

We will use Pulumi to deploy our function app. The Pulumi app creates the function app along with the Key Vault containing the Twilio account token necessary for the API call to send a text message. For more information, see the README file in the infrastructure folder of the source repo.

Once the Pulumi app is deployed, you can get the URL for your function app, in order to complete the IFTTT Applet creation in the next step.

IFTTT Applets

IFTTT allows you to create custom applets, which is basically creating your own recipe of “this” and “that”. To create a new applet, click your avatar in the top-right corner on https://ifttt.com, and click Create.

Click on + This and choose the service called myQ. Most of the garage door openers here in the USA are made by the Chamberlain Group anyway and you are most likely using one of those. All of those openers work with the myQ internet gateway. Your alternative would be to buy a myQ Smart Hub.

Click + That and search for Webhook to select it. You will need the URL of the Function App that was deployed using Pulumi. You can get this URL by navigating to https://portal.azure.com, too. Since the infrastructure was deployed using Pulumi, we can easily fetch its output by running pulumi stack output webhookUrl in the infrastructure folder. We can now complete the Webhook action’s configuration in IFTTT.

Note: Since the function app is exposed to the internet, we don’t want just about anyone to be able to call it. Instead, we will use the built-in function app authorization keys to allow only IFTTT to invoke it. Any other caller without the function key will receive a 401 Unauthorized error due to this auth requirement.

Completing the IFTTT applet creation for webhook action.

Twilio

In order to send a text message, create an account on Twilio and purchase a Programmable SMS number. Your account SID and token can be found on the dashboard page of the Twilio Console or on the Settings page under the API Credentials section.

Final Notes

A few things important things to note:

  • Entity functions (part of Durable Functions 2.x) are a preview feature, though, the Durable Function extensions (1.x) themselves is GA.
  • The KeyVault in the infrastructure is not necessary for a project like this, but it is very easy to create one with Pulumi. And with Azure’s new Managed Identity, it is even easier to configure application access to secrets.
  • To learn more about security best practices on Azure, read this excellent post by Mikhail Shilkov.

In the next post, we will take a closer look at the Pulumi app used to deploy the Azure Function App.

Static sites and Functions

This is the fifth and last part of a series of posts I am writing about building a static site with VueJS. In this post, I will walk-through how you could use Functions-as-a-Service for your next project…or your current too.

Static sites typically don’t get all of the infrastructure attention, that other stuff does. Many developers still think that SPAs, whether static sites or not, need to be hosted with an always-running server infrastructure. It is unnecessary. With the advent of ServiceWorkers and the Progressive Web App movement, you really don’t need a server running all of the time.

Most devs are familiar with using a CMS like WordPress and then buying a domain to serve a website. Most websites don’t need all of that infrastructure. The price is a modest $4/mo according to their pricing page, you just get the basic with the paid plan. Not a big deal, though. But if you wanted to do SEO, custom analytics etc., you are looking at the next pricing tier or perhaps the most expensive one, at $25/mo, if you are looking for a few more knobs/levers to turn.

This is the architecture I used for Net Your Problem.

Static websites architecture with Functions-Page-1
Fig.1 – A simple cloud architecture for SPAs.

I have automated the part where I build the VueJS app and upload it to the Azure Storage account using a PowerShell script (see this gist), which is purely based on the AzureRM PS module.

Great. Now, let’s talk about how these infrastructure systems talk to each other, to cohesively power your next project.

The inner details

A CDN works by aggressively caching static resources (JS, CSS, HTML, PNGs etc.). You can also give a hint to a CDN service to cache additional mime-types by setting the Cache-Control header attribute. A typical CDN service has “edge” nodes all over the world. When you provision a CDN service from any of the cloud providers, you are not choosing the edge nodes in a particular region. CDN services are globally distributed, by default. Each CDN service does their billing differently. For example, I know that the Azure CDN service offers a tiered pricing model based on a group of countries in each tier. So traffic from different countries will be billed at different rates, based on the amount of data transferred from the CDN to the clients (browsers).

As shown in fig.1, the CDN is connect to the Function App, meaning that the CDN will source the static assets from the Function App. But the Function App in turn is connected to a storage account. Technically, all of these can be services from any of the 3 major cloud providers (Azure, AWS, GCP). It doesn’t make sense, though, to create individual services in multiple clouds. You would be charged more for inter-data center data transfer. So it is best to co-locate all of these, except of course, the CDN, which is always global, regardless of whose CDN service you end up using.

The connection between the CDN and the Function App is pretty simple, as it is just a matter of specifying the origin settings for the CDN. The connection between the Function App and the Storage Account requires a little bit more than just specifying the URL. We have to detect the incoming request at the Function and proxy it to the storage account, to let the storage account serve the static asset. Essentially, the Function App serves as a reverse-proxy here for some URL patterns, and for others, as a service that may or may not return a response, i.e., if there is an actual function that can handle the request.

Bonus! Let’s talk about automation

Let’s introduce a CI/CD pipeline into the mix. Azure has a really good all-in-one Ops platform called Azure DevOps, previously known as Visual Studio Team Services or VSTS, and even before that, Visual Studio Online. Anyway, the point of the platform is like Bitbucket or GitHub, where you have everything in one place. CI/CD pipeline integrations, release management, project planning (Kanban as well as sprint-based, whichever you are into), private/public repos, wikis, and also a private feed for your Nuget (.Net packages), npm or maven packages too!

Don’t take my word for it, though. After all, I am just some random programmer on the internet. People say a lot of things. But seriously, you should check it out.

Static websites architecture with Functions-Page-2

Here’s the screenshot of the pipelines page showing you the CI and PROD build pipelines for Net Your Problem.

Screenshot_2018-10-09 Builds - Pipelines

Here’s the CI/CD pipeline in Azure DevOps.

The pipeline is simple. Run npm ci –> npm install –> npm run build –> upload to Az Storage –> store artifacts. That’s it. I have a trigger for this pipeline, which would kick-off this build every time a branch is updated through a merge.

Admittedly, all of this may look like overkill. Trust me, it is not. I spend about a minute to run the build and then to upload the files to the Azure Storage. Then, sometimes, I have to purge the CDN cache, because, well, it works too well sometimes :). Overall, I could spend anywhere between 1-10 mins, and on average ~5 mins, deploying some changes. Now, repeat this several times as I am actively developing something or want to see how things look in Dev, or I want to show something to my ahem client (my girlfriend), the time investment adds-up really quickly. This setup allows me to focus just on the coding part and push up my changes for a quick review on the dev site, before I create a release for the PROD and have approved changes go to the live site immediately. All of this is a pleasant experience for me. To my girlfriend, it makes no difference, and that’s a good thing. She just sees the dev site and the live site. That’s it.

You see, the delays in development, often affects our customers. When I have a pipeline, that works well, my customer doesn’t get affected by it. They just see what they need to see. To them, in the end, it matters if they are able to see what they want and if it works. If the process gets in their way, they simply won’t get it. If I could use the overused automobile world for an analogy. This experience would be akin to taking our car to a shop for an oil change. At the end of it, we just want to drive our car out of the shop with a new oil filter, and fresh oil. We don’t care and most of us don’t want to know how they were able to do an oil change without an appointment. On the other hand, if the oil change took too long, then we want an explanation and all of the shop’s fancy equipment and ISO certifications wouldn’t save them from our negative experience.