The New Full-Stack Dev

There was a time when calling oneself a full-stack dev was met with confused faces. It was a fairly new idea. No one really understood what it really meant and if it was real. It was sort of like gluten and people claiming gluten allergy.

Years later, both couldn’t be more real. In the early days of being a full-stack dev, it meant you could write services powering the UI (aka the backend), as well as some really cool jQuery selectors (like $("#someDiv .items:nth-child(2)") — I’ll let you figure out what that means.), some HTML and some CSS. If you knew jQuery you were the dev to go to at work.

The full-stack developer movement couldn’t have started without javascript templating, backbone.js, and later SASS. Pretty soon after those were introduced a lot more developers claimed to be full-stack developers, which was bolstered by the creation of Twitter’s Bootstrap. And suddenly, there was an explosion in the number of developers who could comfortably write both UI, and backend services. The gates to UI development were knocked-down pretty hard. There was probably a hot new frontend framework every week. From BackboneJS (the OG of bringing Model/View/Collection paradigm to frontend development, in my opinion) to today’s React/Angular/Vue — today’s big 3.

A new framework was announced on JS Weekly!? SWIIITCH!!

Web Developer

From there it was a natural progression for developers to try and tackle writing mobile apps with PhoneGap, which was later renamed Apache Cordova. Arguably one of the most contested ways to write a mobile app. It was nonetheless a way for a web developer to also try their hand at mobile development too. To this day, there are numerous frameworks that promise the “write-once-run-anywhere” type of semantics. Xamarin, Flutter, React Native, NativeScript, Ionic Framework are all popular frameworks in that space. Regardless of what you and I believe the best way to build a mobile app to be, it was yet another skill a full-stack developer could add to their skillset.

Other Developments

Back when cloud services were first introduced, almost no one cared how you got to “the cloud”, so as long as you were “on the cloud”. Scripting languages became uber-popular in the ops area. Knowledge of bash, and PowerShell were (and probably still are) indispensable. Tools like Chef, Puppet invigorated the infrastructure space with their ideas. There are probably more tools that I have probably never even heard of.

AWS has CloudFormation templates, Azure has ARM templates, and Google has…uh, I don’t even know and I don’t want to. This is a clear and present problem.

All this while every major cloud provider was cooking up their own template-based toolset to deploy services to their respective clouds. Surely all of the great minds that thought of the amazing ways to build software for the cloud could have seen the problem a mile away, right? Nope. AWS has CloudFormation templates, Azure has ARM templates, and Google has…uh, I don’t even know and I don’t want to. This is a clear and present problem. It is what kept the infrastructure space away from the general reach of your average developer who just wants to deploy a simple service to the cloud.

Redefining Full-Stack Dev

With HashiCorp’s Terraform, this changed. Infrastructure “gurus” started to see the need for a system that would let them deploy repeatable, and predictable infrastructure to the cloud. This works really well if this is what you want. But if you don’t want to learn a new DSL?

If you noticed at the beginning of this post, every innovative piece of technology introduced in the developer ecosystem, involved taking something that was accessible to a niche of developers and turning it into a programmable thing. HTML became JS template, CSS was too brittle, so SASS was created which brought some general-purpose programming language paradigms to CSS (when was the last time you wrote CSS? I mean, raw CSS, not SCSS as you know it today.), mobile apps could be built frontend frameworks. At each stage, the adoption of the respective technology was through-the-roof.

We have been going about trying to solve the cloud infrastructure tooling the wrong way all this time.

Allowing developers to do what they do best — write software, will certainly bring infrastructure to the masses. This is why tools like Pulumi will disrupt this space. The movement to bring infrastructure to the masses has begun. Pulumi brings the programmability, the “coding” aspect to infrastructure. For the first time, it really feels like I can truly program the cloud.

That is not to say the tools that have been created so far have no place in building infrastructure. Nope. Quite the opposite. I think there is a need for these tools. For instance, not everyone is using the cloud the way cloud providers want us to use them. Many of them are tied to legacy on-premise systems, leading to the so-called “Hybrid Environments”. Whatever the setup, there is a place for each of those tools.

And for those of us building something new for the cloud, there’s Pulumi.

So let me define the new full-stack dev. A developer who can work on all tiers of an application, including the infrastructure.

Infrastructure with DotNet Core 3.0

Using Pulumi to automate home automation recipes

In a previous two-part blog series I wrote about building a home automation recipe and automating its deployment with a Pulumi app. That was written in TypeScript. Now that Pulumi supports DotNet Core 3.0 with Pulumi CLI 1.5.0+, I ported over the infrastructure to C# as well, and added it as a C# project to my existing Solution.

A C# app and a C# infrastructure app. Nice. It was worth the wait.

What do you think would have happened when I ran pulumi up in the C# version of the same infrastructure? You wouldn’t expect any changes since the infrastructure wasn’t changed — just the language that it was written in, right? Well, that’s exactly what happened (or didn’t happen, depending on how you look at it.) Nothing changed when I ran a pulumi preview after I completed porting over the application.

That, my friends, is the power of programming languages…and proper infrastructure as code.

Part 2: Building Home Automation Recipes with Pulumi

Pulumi

In the previous post, I walked-through a home automation recipe where I showed you how easy it is to wire-up existing automation platform with your own custom solution that runs in the cloud. Let’s take a look at how to deploy that on Azure using…code — not templates.

Getting Ready

Project Creation

Create an empty directory somewhere on your local disk. You can init a git repo as well if you’d like, but not necessary for using Pulumi for this project. If you are looking to use Pulumi at work, you should definitely use an SCM of some kind.

Open your favorite terminal window depending on your OS, and cd to your newly-created directory, then run pulumi new (assuming pulumi is on your PATH).

This will show you a few templates, that you can use to create a project. You don’t need to use any of them. In fact, you don’t even have to run pulumi new. You can do this all manually if you prefer that, but it’s certainly the easiest way to get started. Let’s select the azure-typescript template.

Run pulumi new in a new empty directory.

In the screencast snippet above I accepted the defaults for everything. Feel free to enter your own values. You can see that the screencast ends with Pulumi installing the dependencies. Pulumi will create a minimal TS-based app with the required npm dependencies for Azure and TypeScript (since I chose the azure-typescript template).

Imports And Config

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";
import * as path from "path";

Let’s take a look at the imports at the top of the index.ts file. All pulumi npm packages will be under @pulumi.

@pulumi/pulumi is the core Pulumi SDK providing things like the configuration class, the programming model constructs for things components and custom resources. It also contains the types for recognizing inputs and outputs, and some helpers to work with both of those.

@pulumi/azure is the Azure-specific SDK for creating Azure resources.

path is the standard NodeJS package.

import { buildFunctionsProject} from "./projectBuilder";

projectBuilder is a TypeScript file that exports just one function buildFunctionsProject.

const namePrefix = "grge-mon";
const config = new pulumi.Config();

The Config class is used to retrieve config key/values stored in the Pulumi.<stack_name>.yaml file. Learn more here.

const twilioAccountToken = config.requireSecret("twilioAccountToken");

This is a really cool part. You can add secrets to your config and retrieve them easily in your code. Pulumi will track the value of this variable as a secret. You add secrets to your stack config using pulumi config --secret set <key> <value>. Learn more here.

buildFunctionsProject(path.join("..", "GarageDoorMonitor"));

As mentioned before, the buildFunctionsProject is an exported function used from another file. This function builds our .NET Core Functions project.

Creating The Infrastructure

Alright. Let’s create some Azure resources.

const resourceGroup = new azure.core.ResourceGroup(`${namePrefix}-group`);

First, we will need a resource group to put our resources into. This is an Azure construct and not a Pulumi-specific thing. If you have worked with Azure, everything you know about it applies even when you use a programming language in Pulumi.

Azure KeyVault

const kv = new azure.keyvault.KeyVault(`${namePrefix}-vault`, {
    resourceGroupName: resourceGroup.name,
    skuName: "standard",
    tenantId: azure.config.tenantId!,
    accessPolicies: [{
        tenantId: azure.config.tenantId!,
        // The current principal has to be granted permissions to Key Vault so that it can actually add and then remove
        // secrets to/from the Key Vault. Otherwise, 'pulumi up' and 'pulumi destroy' operations will fail.
        //
        // NOTE: This object ID value is NOT what you see in the Azure AD's App Registration screen.
        // Run `az ad sp show` from the Azure CLI to list the correct Object ID to use here.
        objectId: "your-SP-object-ID",
        secretPermissions: ["delete", "get", "list", "set"],
    }],
});

We create the KeyVault resource using the new operator. This sort of looks like the properties in the ARM template for creating a KeyVault, except, here you get the advantage of strongly-typed arguments. This makes it really easy to specify values without second-guessing what you need to provide to a property.

Note: A KeyVault resource in Azure uses Access Policies to restrict who can administer it. This also means that if you use a service principal (or your own personal account) to run the Pulumi app, it will need access to the KeyVault to add/remove secrets/keys/certificates. This is why we specify the object id of the account in the access policies initially.

Adding a secret

const twilioSecret = new azure.keyvault.Secret(`${namePrefix}-twil`, {
    keyVaultId: kv.id,
    value: twilioAccountToken,
});

Let’s add a secret to the KeyVault. Notice how we specified the KeyVault to which the secret should be added by simply referencing the variable kv from the previous step. That’s how you would normally pass values to anything that depends on a value from another object, right? But why am I calling this out like it’s a big deal? Remember that this is no ordinary app. We are dealing with infrastructure resources here. These are not just some variables with values stored in memory.

These variables represent actual resources on Azure. This also means that when the KeyVault is still being created, a secret cannot be added to it. So how does Pulumi know when to extract the id property from it? Well, the answer to that is resource ordering. Pulumi knows that the resource represented by kv needs to finish creating before a new Secret resource is added to it. This is similar to how you would specify dependsOn in an ARM template to tell ARM how to order your resources and to flow outputs of one resource’s creation to another as an input. Instead, in Pulumi this happens automatically as you just go about writing regular TypeScript code.

String interpolation with infrastructure resource outputs

const twilioSecretUri = pulumi.interpolate`${twilioSecret.vaultUri}secrets/${twilioSecret.name}/${twilioSecret.version}`;

String interpolation in modern JavaScript (and TypeScript) is achieved by enclosing a string using the backtick character and using ${} to insert a variable. But here we are dealing with special resources. The variables used in the string format are not (yet) present. This means it cannot be evaluated yet, or else we would get undefinedsecrets/undefined/undefined as the value of twilioSecretUri. This is why the Pulumi SDK provides pulumi.interpolate. Use that with the standard JS interpolation characters and Pulumi will recognize that the interpolation must be considered for resource availability before evaluation.

Creating an app insights dashboard is a piece of cake

const appInsights = new azure.appinsights.Insights(`${namePrefix}-ai`, {
    applicationType: "web",
    resourceGroupName: resourceGroup.name,
});

There is not much to say here (and that’s a good thing here!) The code is pretty self-explanatory.

Function App

const durableFunctionApp = new azure.appservice.ArchiveFunctionApp(`${namePrefix}-funcs`, {
    resourceGroup,
    archive: new pulumi.asset.FileArchive("../GarageDoorMonitor/bin/Debug/netcoreapp2.1/publish"),
    appSettings: {
        "runtime": "dotnet",
        "TwilioAccountToken": pulumi.interpolate`@Microsoft.KeyVault(SecretUri=${twilioSecretUri})`,
        "APPINSIGHTS_INSTRUMENTATIONKEY": pulumi.interpolate`${appInsights.instrumentationKey}`,
        "TimerDelayMinutes": config.getNumber("timerDelayMinutes") || 2,
    },
    httpsOnly: true,
    identity: {
        type: "SystemAssigned"
    }
});

// Now that the app is created, update the access policies of the keyvault and
// grant the principalId of the function app access to the vault.
const principalId = durableFunctionApp.functionApp.identity.apply(id => id.principalId);

Pulumi provides some higher-level helpers for well-known/popular resources such as Azure Functions to ease with the package/deployment. Normally you will have to package your code as a zip and deploy it to the Function App. You can do this using a built-in task extension in Azure DevOps, or you can manually zip up your functions and deploy them directly on Azure. Using Pulumi to do this is very easy. And because Pulumi tracks every resource, it will only trigger an update to the code package if you truly changed your functions code. Otherwise, nothing is changed.

KeyVault access

// Grant App Service access to KV secrets
const appAccessPolicy = new azure.keyvault.AccessPolicy(`${namePrefix}-app-policy`, {
   keyVaultId: kv.id,
   tenantId: azure.config.tenantId!,
   objectId: principalId,
   secretPermissions: ["get"],
}, { dependsOn: durableFunctionApp });

Our Function App needs access to the KeyVault to access the secret. So let’s create a new access policy and attach it to the KeyVault.

Outputs

export const webhookUrl = durableFunctionApp.endpoint;

At a basic level, think of your Pulumi app as a thing that is creating several things, but there is typically some output that is of interest to you for your application code. For example, the URL for the API service, IP address of a loadbalancer, the hostname of a managed Cosmos DB instance etc. To create an output you simply need to export it.

You can retrieve outputs from your stack later if you would like using the Pulumi CLI by running pulumi stack output <outputName> where <outputName> is the name of the variable that you exported.

Outputs play an important role in making your infrastructure modular. In this post we only used a single stack. In a more practical scenario, you are perhaps working with multiple teams and each of them may want to have their own stack. But you may have a dependency on the output of one of those stacks, you can use outputs from another stack in your own stack using a StackReference.

Pulumi Console And The Managed Backend

Just like the Azure Portal, Pulumi has a Console UI. The Console gives you a detailed view of all the resources in each stack, their outputs, the timeline of events, activities etc.

Pulumi tracks the state of your resources. It shows you diffs from the current state as you make changes to your infrastructure. At the beginning of this post I stated I would explain why you should sign-up for an account. To get the state tracking and diffs you don’t need an account on Pulumi Console. However, if you want to keep that state safe, highly-available, and make sure concurrent updates are not performed on your infrastructure, which you will need when you are developing for production, you should use the Pulumi-managed backend, and let it take care of all of that for you. The alternative to this is, to manage all that on your own. Learn more about that here.

Closing Notes

  • It is important to note that what you saw above is not a Pulumi-flavor of TS. It is just TS. The same TS you would use to develop Angular apps or whatever else you use TS for these days. This also means everything that the language provides is available for you to use. There are no restrictions, other than what the Pulumi resource provider inflicts on your app.

  • Review the Pulumi Programming Model page for some advanced concepts. Particularly, the Components.

  • For JS and TS-based Pulumi apps, the runtime is NodeJS. So each TS file is transpiled to JS and executed inside the Node runtime just like any other Node app. This also means that you can use just about any Node-compatible npm package, including the built-in Node packages as well. In fact, I used the child_process package to execute the dot net publish command.

  • If you are planning to use Pulumi on Azure DevOps, then checkout Pulumi’s free Task Extension for build and release definitions.

Building Home Automation Recipes with Pulumi

Using Pulumi to automate home automation recipes

Home automation is easier than ever with a plethora of IoT-connected devices enabling everyday appliances in your home to be internet-enabled. When you write a custom piece of integration for your IoT, often times deployment to the cloud becomes an afterthought, only to become a nightmare when you actually want to update it later, and you don’t remember where/how you deployed it. With Pulumi, you don’t have to worry about that anymore. You can develop your IoT integration app, as well as your program your infrastructure as an app.

The Garage Door Opener

Most people have an internet hub connected to their automatic garage door opener, which allows them to remotely monitor the garage door as well as open/close it using a mobile app. But what about when you forget to close it and it stays open? Neither the app nor the existing home automation recipes on the home automation website IFTTT have a way to remind you that you left it open. To solve this problem I tried not to build something of my own, but instead try to use Zapier, a task automation platform.

Note: The source code for this post is available here.

The First Attempt

My first attempt involved using Zapier. It would have worked if there was a way to update a state while waiting for a timer to fire. I used IFTTT to connect the myQ service to fire a Webhook request each time the garage door opened or closed. The webhook receiver was a Zapier webhook “catch” action, which I then connected to a timer delay before sending me a text message via Twilio. It mostly worked, except, if I closed the garage door before the timer fires, there was no way for me to update the state and therefore, cancel sending the text message.

Here’s the “zap” I ended up creating:

A Zap on Zapier using built-in actions.

Durable Functions on Azure Functions

Durable Functions is an extension to the already popular Azure Functions platform. This means, you can write functions with an external trigger (HTTP, Queue etc.) and have it trigger an orchestration. Each orchestration instance is automatically tracked by the platform. Checkout the API reference to see what you can control about an orchestration instance.

Durable Function Types

There are other durable function types, learn more about them here. The following are just the types used in this project.

Orchestration Functions

Each function has a trigger type that identifies _how_ that function can be triggered. Orchestration functions are no different. Orchestration functions typically don’t do any work other than, you guessed it, orchestrate other functions that do the work.

Activity Functions

Activity functions are responsible for most of the work in an orchestration. You can make HTTP calls, call other activity functions etc.

Entity Functions

Entity functions are only available as part of the Durable Functions 2.x beta. This is in public preview.

Entity functions allow you to represent your orchestration instance with a state. It is up to you on whether each orchestration instance has its own entity or if your state is a singleton. This is controlled by the way that entities are persisted. Each entity is made up of two components:

  • An entity name: a name that identifies the type of the entity (for example, “Counter”).
  • An entity key: a string that uniquely identifies the entity among all other entities of the same name (for example, a GUID).

IFTTT + Azure Durable Functions + Twilio + Pulumi

This is the high-level view of the solution I finally ended up with.

  • IFTTT receives signals from the garage door opener.
  • IFTTT then calls the function app.
  • The function app waits for couple of minutes and if the door still isn’t closed by then, sends a text message using Twilio.

Function App

The only external trigger in the function app is the HTTP trigger used in this function. An orchestration instance is created only through the orchestration client, injected into the HTTP-triggered function as a param.

[FunctionName("Main")]
public static async Task RunOrchestrator(
    [OrchestrationTrigger] IDurableOrchestrationContext ctx,
    ILogger log)
{
    var delay = Environment.GetEnvironmentVariable("TimerDelayMinutes");
    var delayTimeSpan = TimeSpan.FromMinutes(Convert.ToInt32(delay));
    DateTime timer = ctx.CurrentUtcDateTime.Add(delayTimeSpan);
    log.LogInformation($"Setting timer to expire at {timer.ToLocalTime().ToString()}");
    await ctx.CreateTimer(timer, CancellationToken.None);

    try
    {
        // The use of a critical block, though optional, is recommended here.
        // Updates to durable entities are serial, by default.
        // Having the lock ensures that the entity state we are reading is guaranteed to
        // be the current value of the entity.
        using (await ctx.LockAsync(EntityId))
        {
            var currentState = await ctx.CallEntityAsync<string>(EntityId, "read", null);
            log.LogInformation($"Current state is {currentState}.");
            // If the door is closed already, then don't do anything.
            if (currentState.ToLowerInvariant() == "closed")
            {
                log.LogInformation("Looks like the door was already closed. Will skip sending text message.");
                return;
            }
            await ctx.CallActivityAsync("SendTextMessage", null);
        }
    }
    catch (LockingRulesViolationException ex)
    {
        log.LogError(ex, "Failed to lock/call the entity.");
    }
    catch (Exception ex)
    {
        log.LogError(ex, "Unexpected exception occurred.");
    }
}

Deploying the Infrastructure using Pulumi

We will use Pulumi to deploy our function app. The Pulumi app creates the function app along with the Key Vault containing the Twilio account token necessary for the API call to send a text message. For more information, see the README file in the infrastructure folder of the source repo.

Once the Pulumi app is deployed, you can get the URL for your function app, in order to complete the IFTTT Applet creation in the next step.

IFTTT Applets

IFTTT allows you to create custom applets, which is basically creating your own recipe of “this” and “that”. To create a new applet, click your avatar in the top-right corner on https://ifttt.com, and click Create.

Click on + This and choose the service called myQ. Most of the garage door openers here in the USA are made by the Chamberlain Group anyway and you are most likely using one of those. All of those openers work with the myQ internet gateway. Your alternative would be to buy a myQ Smart Hub.

Click + That and search for Webhook to select it. You will need the URL of the Function App that was deployed using Pulumi. You can get this URL by navigating to https://portal.azure.com, too. Since the infrastructure was deployed using Pulumi, we can easily fetch its output by running pulumi stack output webhookUrl in the infrastructure folder. We can now complete the Webhook action’s configuration in IFTTT.

Note: Since the function app is exposed to the internet, we don’t want just about anyone to be able to call it. Instead, we will use the built-in function app authorization keys to allow only IFTTT to invoke it. Any other caller without the function key will receive a 401 Unauthorized error due to this auth requirement.

Completing the IFTTT applet creation for webhook action.

Twilio

In order to send a text message, create an account on Twilio and purchase a Programmable SMS number. Your account SID and token can be found on the dashboard page of the Twilio Console or on the Settings page under the API Credentials section.

Final Notes

A few things important things to note:

  • Entity functions (part of Durable Functions 2.x) are a preview feature, though, the Durable Function extensions (1.x) themselves is GA.
  • The KeyVault in the infrastructure is not necessary for a project like this, but it is very easy to create one with Pulumi. And with Azure’s new Managed Identity, it is even easier to configure application access to secrets.
  • To learn more about security best practices on Azure, read this excellent post by Mikhail Shilkov.

In the next post, we will take a closer look at the Pulumi app used to deploy the Azure Function App.

Static sites and Functions

This is the fifth and last part of a series of posts I am writing about building a static site with VueJS. In this post, I will walk-through how you could use Functions-as-a-Service for your next project…or your current too.

Static sites typically don’t get all of the infrastructure attention, that other stuff does. Many developers still think that SPAs, whether static sites or not, need to be hosted with an always-running server infrastructure. It is unnecessary. With the advent of ServiceWorkers and the Progressive Web App movement, you really don’t need a server running all of the time.

Most devs are familiar with using a CMS like WordPress and then buying a domain to serve a website. Most websites don’t need all of that infrastructure. The price is a modest $4/mo according to their pricing page, you just get the basic with the paid plan. Not a big deal, though. But if you wanted to do SEO, custom analytics etc., you are looking at the next pricing tier or perhaps the most expensive one, at $25/mo, if you are looking for a few more knobs/levers to turn.

This is the architecture I used for Net Your Problem.

Static websites architecture with Functions-Page-1
Fig.1 – A simple cloud architecture for SPAs.

I have automated the part where I build the VueJS app and upload it to the Azure Storage account using a PowerShell script (see this gist), which is purely based on the AzureRM PS module.

Great. Now, let’s talk about how these infrastructure systems talk to each other, to cohesively power your next project.

The inner details

A CDN works by aggressively caching static resources (JS, CSS, HTML, PNGs etc.). You can also give a hint to a CDN service to cache additional mime-types by setting the Cache-Control header attribute. A typical CDN service has “edge” nodes all over the world. When you provision a CDN service from any of the cloud providers, you are not choosing the edge nodes in a particular region. CDN services are globally distributed, by default. Each CDN service does their billing differently. For example, I know that the Azure CDN service offers a tiered pricing model based on a group of countries in each tier. So traffic from different countries will be billed at different rates, based on the amount of data transferred from the CDN to the clients (browsers).

As shown in fig.1, the CDN is connect to the Function App, meaning that the CDN will source the static assets from the Function App. But the Function App in turn is connected to a storage account. Technically, all of these can be services from any of the 3 major cloud providers (Azure, AWS, GCP). It doesn’t make sense, though, to create individual services in multiple clouds. You would be charged more for inter-data center data transfer. So it is best to co-locate all of these, except of course, the CDN, which is always global, regardless of whose CDN service you end up using.

The connection between the CDN and the Function App is pretty simple, as it is just a matter of specifying the origin settings for the CDN. The connection between the Function App and the Storage Account requires a little bit more than just specifying the URL. We have to detect the incoming request at the Function and proxy it to the storage account, to let the storage account serve the static asset. Essentially, the Function App serves as a reverse-proxy here for some URL patterns, and for others, as a service that may or may not return a response, i.e., if there is an actual function that can handle the request.

Bonus! Let’s talk about automation

Let’s introduce a CI/CD pipeline into the mix. Azure has a really good all-in-one Ops platform called Azure DevOps, previously known as Visual Studio Team Services or VSTS, and even before that, Visual Studio Online. Anyway, the point of the platform is like Bitbucket or GitHub, where you have everything in one place. CI/CD pipeline integrations, release management, project planning (Kanban as well as sprint-based, whichever you are into), private/public repos, wikis, and also a private feed for your Nuget (.Net packages), npm or maven packages too!

Don’t take my word for it, though. After all, I am just some random programmer on the internet. People say a lot of things. But seriously, you should check it out.

Static websites architecture with Functions-Page-2

Here’s the screenshot of the pipelines page showing you the CI and PROD build pipelines for Net Your Problem.

Screenshot_2018-10-09 Builds - Pipelines

Here’s the CI/CD pipeline in Azure DevOps.

The pipeline is simple. Run npm ci –> npm install –> npm run build –> upload to Az Storage –> store artifacts. That’s it. I have a trigger for this pipeline, which would kick-off this build every time a branch is updated through a merge.

Admittedly, all of this may look like overkill. Trust me, it is not. I spend about a minute to run the build and then to upload the files to the Azure Storage. Then, sometimes, I have to purge the CDN cache, because, well, it works too well sometimes :). Overall, I could spend anywhere between 1-10 mins, and on average ~5 mins, deploying some changes. Now, repeat this several times as I am actively developing something or want to see how things look in Dev, or I want to show something to my ahem client (my girlfriend), the time investment adds-up really quickly. This setup allows me to focus just on the coding part and push up my changes for a quick review on the dev site, before I create a release for the PROD and have approved changes go to the live site immediately. All of this is a pleasant experience for me. To my girlfriend, it makes no difference, and that’s a good thing. She just sees the dev site and the live site. That’s it.

You see, the delays in development, often affects our customers. When I have a pipeline, that works well, my customer doesn’t get affected by it. They just see what they need to see. To them, in the end, it matters if they are able to see what they want and if it works. If the process gets in their way, they simply won’t get it. If I could use the overused automobile world for an analogy. This experience would be akin to taking our car to a shop for an oil change. At the end of it, we just want to drive our car out of the shop with a new oil filter, and fresh oil. We don’t care and most of us don’t want to know how they were able to do an oil change without an appointment. On the other hand, if the oil change took too long, then we want an explanation and all of the shop’s fancy equipment and ISO certifications wouldn’t save them from our negative experience.

Data-driven sites

This is the fourth part in a 5-part series about building a static site using VueJS. In this part, I’ll show you an example of how I built Net Your Problem by thinking of static content as data, rather than…well, static content.

We’ll go through this article looking at one specific section of Net Your Problem – the Projects section. There are two cards, with a title, a cover image for the card, and a button that opens a modal dialog.

Screenshot_2018-10-09 Net Your Problem
The Projects section of Net Your Problem.

Let’s look at the template for this,

There are two projects shown on the page, the template only shows one <card> tag with some data-bindings, along with some code to track some events with Google Analytics. Nothing too crazy here. But where is all of the content coming from? Netlify CMS.

If you think of your site as simply the presentation layer, that needs to serve content, think of a CMS as the database that stores your data, i.e., content. Netlify is a bit special. It actually doesn’t use any database, well, technically no. But one could argue that its use of a version-controlled filesystem is like a database. After all, a database has files too. Anyway, back to Netlify and how it works. Netlify basically provides a content editing platform on top of popular git version control systems like GitHub, and BitBucket. You can read about them here.

What I have done for Net Your Problem is, use Netlify CMS as the content editing platform, almost like WordPress. I italicized “almost” because, although WordPress is a CMS too. It differs in many ways. Well, first major difference is that, WordPress uses databases. You also need to host your content on their platform. On top of that, the articles, very much like this one, can only be published to a sub-domain of their own domain, at least under the Free plan. You can install WordPress on your own servers if you are adventurous and want to deal with all of the jazz of the setup and maintenance.

The thing with Netlify is, that the content simply gets stored as Markdown, JSON, or TOML files in your favorite version control SaaS platform. The files are organized in directory structures, that makes it easy for you to read the files. You can simply make ajax calls using the public APIs for GitHub or Bitbucket. The downside (if at all!) is, that you have to make your repository public in order to be able to call it anonymously, i.e., without authentication, from your website’s JS.

Let’s look at the code to fetch the content, which Netlify stores in our content repository. The gist below shows the script portion of the same Projects component, for which we saw the <template> portion above.

A few things to note,

  • axios is used as the HTTP client library
  • There are two API calls
    • GET the list of projects to show the cards.
    • GET the full content for a project when the user clicks on the READ MORE button in the card.

Once the content is downloaded from the JSON file, which is stored in the Bitbucket repo, I just update the data property that has a template binding attached to it and since the content is stored as a JSON file, the response from the Bitbucket API is…yep, JSON. This link will show you the response for the Projects lists JSON file.

And here’s the modal dialog that shows the full content of a “project” from Net Your Problem. The content itself is a markdown string, which is fed to the <vue-markdown> component, which you can see in the template above.

Screenshot_2018-10-09 Net Your Problem(1)
A modal dialog showing the content for one of the “projects” on Net Your Problem.

We just looked at one section in the site, but I am happy to report that 100% (…ok ok 99.9%..the header navigation is hard-coded) of the site is built this way. At first, all of the content was hard-coded in the site, and I slowly started to convert each component to be completely driven by data fetched through the APIs.

Convinced? Head over to the Netlify CMS docs to get started.

Components in a static site too

In the 2nd part of this series, we learned about some basics of rendering a view. We ended that topic by having a look at the router. The router just had one path, the root (/), which was mapped to a single component called the HelloComponent.

Here’s what we’ll do in this post:

  • Examine the HelloComponent
  • Add a new route, and a new component to handle the route
  • Add a nav link to take the user to the new route
  • Render another component inline without routing

HelloComponent

As you can see, the HTML code for most of what you see when you navigate to http://localhost:8080 comes from this file. So how does Vue know where to render the contents of this component? If you recall, the App.vue, the parent component, has a tag called and I mentioned that this is the output for any component that is routed to by the router.

Add a new route, and a new component to handle the route

Create a new file under the components/ folder. Let’s call it TestComponent.vue. And paste the following content.

Yes, I realize it doesn’t do much. I wanted to show you what a basic component looks like. It doesn’t need to have anything more than just that. With just that much content, you get a valid component. You should be able to imagine now, how easy it is to create a component and have your entire site split into pieces (components), that all come together eventually in the browser.

But wait. We are not done adding the component. We just added a file called TestComponent.vue but we haven’t really used it anywhere. So let’s add a new route to the router/index.js file.

Your router should now look like this:

We just added a new route to the router called /test, which routes to the newly-created TestComponent. Let’s test this out by going to: http://localhost:8080/#/test. You should see most of the content replaced with just the word “Test”. This means our new component has rendered. Great. We confirmed our new component to be working by manually going to the /test route.

Add a nav link to take the user to the new route

Let’s look at adding a route-link so that the user can navigate to this newly-created component of ours. Update the App.vue with this. Somewhere inside the <template> tag, add this markup

<router-link to="/test">Go to Test</router-link>

Vue will take care of the rest. It will render the router-link as an anchor (<a>) tag. You could do this programmatically too, if you don’t want to use an anchor tag. Refresh your browser and you should see a Go to Test link on your page. Click it. You should now see the contents of the TestComponent. That was it.

That’s it. We just learned how to use components in Vue to compose our app of little pieces, which are building blocks for a larger site or web app. I highly recommend reading more about VueRouter here.

Render another component inline without routing

So we saw how we could link to a custom component. What if we simply want to render another component inline, in the context of a parent component? Yep. You can do that too.

To do that, let’s first remove the <router-link> tag, and update your App.vue to this:

Then, go to http://localhost:8080 and you should see the contents of the TestComponent rendered inside the contents of App.vue, which also happens to consist of the contents from HelloComponent. So instead of replacing the contents from HelloComponent, we just augmented the contents of our new component in there.

Since you are rendering the component inline, there is no need for that new route, that we added to the router/index.js. You can remove the /test route from it as well, if you’d like.

Using machine learning to classify NSFW images

Hi, all! I have a favor to ask. I have an alpha version of my app, Transitions, that can classify images as NSFW with an 80% or higher accuracy, most of the times (top-1 accuracy as it is known in the ML world). It uses Mobilenet V1. The predictions are made on-device. So your data is not used for predicting whether an image is NSFW or not.

I was wondering if any of you would be interested in signing-up to be in an Internal Test Track? I already run an Open Beta program but before releasing a beta, I’d like to test it with a closed loop of users. Sorry, but my app is Android-only. You’ll need a device with Android 6.0 or higher. So please PM me if you are interested. Once you are part of the testing program, I am hoping you would also be OK to submit feedback so I can improve the model’s accuracy.

Why NSFW classification?

Transitions serves user-generated content from flickr, 500px, and Unsplash. Even with category filtering, inappropriate images show up all the time. I currently have an ESRB rating of Teen on Google Play, because of this. I can’t change that rating;  even with the automated classification and dynamic blurring of such images. But I want to give my users the ability to browse high-quality photos on Transitions without the uncertainty of what they’ll see next.

What do you (as a tester) get from this?

Uh, the joy of helping me? Early access? Take your pick.. 😀

VueJS basics

his is the second part in a series of articles I am writing about how I used VueJS to build a static site.

Assumptions for this article

  • You have NodeJS 8.x installed with npm.
    • You can verify your installation by running these commands in a PowerShell window,
node --version
npm --version
  • You have some knowledge of CLI-based bootstrapping, and development.
  • You are using some sort of a web development IDE. I highly recommend VS Code.

Setup

You’ll want to get the official Vue CLI first. Be sure to install the latest stable release of the CLI. There is a beta version out there. At the time of this writing, 3.0 is still in beta so you can refer to the README for the 2.x version on the master branch. Then, run:

npm install -g vue-cli
vue init webpack my-project

Go through the prompts in your shell and complete the boilerplate project setup. These are my settings:

? Project name static-site
? Project description A Vue.js project
? Author Blog Author <author@blog.com>
? Vue build runtime
? Install vue-router? Yes
? Use ESLint to lint your code? Yes
? Pick an ESLint preset Standard
? Set up unit tests Yes
? Pick a test runner karma
? Setup e2e tests with Nightwatch? Yes
? Should we run `npm install` for you after the project has been created? (recommended) npm

You’ll notice that I used the word webpack in the command to create the project. That’s the name of the webpack template in the CLI. The CLI will automatically create the necessary webpack base configuration with all the loaders, extractors etc.
Once you are done with the setup, you can start-up your dev server for local development using:

npm run dev

You can access the local server using http://localhost:8080. This is it for setup. For most static sites, I don’t anticipate any changes in the base webpack config. So you don’t have to meddle with it at all, not even when you are ready to build a distribution package.

Basics

The sweet spot, in my opinion, for developing using Vue is, its single-file components. Everything that a component needs, the HTML template, the script, and the CSS (or SCSS, if you are living in the 21st century), is in the same file. This encourages you to split your site (or app) into simpler components that do only one thing. It encourages you to use the parent-child communication pattern effectively by way of props.

I won’t go over everything from the excellent documentation on VueJS’ docs site. But I will go over some of the basics, that I think are important before I explain how I built Net Your Problem.

Declarative Rendering

This is just a fancy way of saying, HTML in; HTML out. There is no special language for you to learn here. Of course, this doesn’t do much other than render the view (the HTML).

Open the App.vue file from the project you setup using the CLI and look at the content within the enclosing <template> tag.

and then in main.js:

All this is doing is, creating a new instance of a Vue app, and then binding the app instance to the root view, which is an element with the ID value “app”. As you can see, in App.vue, this is our <div> tag.

OK. But wait. How did Vue know where to look for an HTML element inside the App.vue component. Take a look at the “render” property in the statement for creating a new instance of a Vue app. Its value is an arrow function that accepts a callback function as a parameter. The callback function expects a Vue component to be passed to it. This render function is what kicks-off the render-ing of all child components when you start composing components from the parent component.

Data binding, conditional rendering and loops

A very important building block for any site, application. I am pretty sure almost every single site, web app out there uses conditional rendering and repeated elements all over.

Let’s modify the App.vue like this:

Now, reload the page. You should see the message Hello, World! right below the logo. If you change the value of the showMessage property from true to false, then you should see the message disappear. This was made possible because of the directive v-show. You can read more about conditional rendering and the other directive called v-if. There is an important performance difference between the two.

Let’s add a repeated element. Modify the App.vue’s template contents like this:

If the page hasn’t refreshed, refresh it, and you should see the additions. The condition items.length > 0 is satisfied since the items array has a non-zero size.  We only used a single <li> tag in the template, however, the final result has two of those. That’s because of the v-for directive. You can read more about it in the list rendering section of the guide.

Routers

Web sites, and web apps use URLs to send the user to new sections of a page, or to completely different pages. Routers are used to load/unload components as the user navigates your site/app. Depending on how you structure your site, you may or may not want to use a router at all. It isn’t necessary if you don’t have complex sections in your site, and you don’t need to orchestrate showing and hiding these sections appropriately.

But let’s walk-through it real-quick. If you recall in App.vue, there is a tag called <router-view/>. This is a special tag, which Vue recognizes as the output of a user navigation. That is, when the user navigates somewhere, the Vue system places the content of the new destination in its place.

Have a look at the router/index.js

That is it for some of the basics. I highly recommend reviewing the introduction to Vue’s concepts. You can read about them here.

Building a static site with VueJS

This post is the first part of a series of articles. The project that I will focus on will be the site I recently developed for Net Your Problem; a company that connects fishermen with companies that recycle old and worn-out nets, in order to produce plastic pellets, which can be turned into new plastic products. If you are a fisherman, or a company that recycles nets, I highly recommend contacting Net Your Problem. I am not just recommending them, because I have a personal connection with the founder of the company. I truly believe in what they are doing.

I should also mention that since the time I built the site, the creator of VueJS announced a new framework, called VuePress, specifically for building static sites using VueJS. You should check it out to decide if it fits your needs. But this is out of scope for my series and I will not be discussing the differences or its features in this series.

Now that we got that out-of-the way. Here’s what I have planned for this series:

  1. Why VueJS
  2. VueJS basics
  3. Components in a static site too
  4. Data-driven sites
  5. Static sites and Functions

Why VueJS?

Why not abc, or xyz, or <insert framework here>?

For a small site that sees little traffic, with not a lot of functionality, I couldn’t justify wanting to use a heavier framework such as Angular or React. I realize React isn’t a framework but a library. In fact, VueJS borrows concepts from both of those technologies. The core team behind Vue has written this article to compare with other frameworks/libraries. I realize that I cannot do justice to every single framework out there. So let me put this out there. If you feel that I have not mentioned a framework that you are using, feel free to comment on this post and I’ll be happy to explore it. I love learning new things.

You should decide the best technology for yourself (or your team) based on what you think works for you, and the problem that you are solving. I am not recommending that you use VueJS for everything, but you should carefully evaluate them and understand why they exist. Here are some pointers for choosing a technology for your next project:

  • Do you already know how to use it?
  • Have you evaluated other frameworks with a small proof-of-concept?
  • How much time did it take to build the proof-of-concept?
  • Can you come up with a reason (or several reasons) as to why you should not use a particular technology?
  • Does the rest of your team know how to use it?
  • Does the

I spent a lot of time looking at a class of micro-frameworks. I even built the first version of the site using nothing but just plain HTML and used Parcel as my build tool. It worked great to start-off. But I quickly found myself wanting to do things outside of the boundaries of Parcel. Instead of fighting the framework, I made the decision to spend the time and switch to something else. As an alternative, I looked at Svelte, Stimulus, and even Hyperapp. These two are very different frameworks. I won’t go deep into how they work but they take a different approach to building web-apps than the big three (that I know), i.e., React, Angular, and Vue.

I had already used VueJS for a prior project. Although, that prior project was not a static site. It was a fully-dynamic web application and I loved using VueJS. Everything just fell into place when I used the framework. So with that good prior experience, I decided to use VueJS. I knew it will serve the purpose extremely well, and also have plenty of room for growth, should there be a need for it. Here’s why I enjoy using Vue:

  • Single-file components.
  • Component-scoped SCSS (although many frameworks offer this).
  • Automatic webpack build config (again, many frameworks offer this).
  • Easy-to-learn, (and re-learn when you are away from it for a long time) concepts.
  • Does not force some choices on you.
    • For example, there is no HTTP client available in Vue. The most popular one to use is axios. There is also vue-resource but, it has since been deprecated in favor of axios.
  • Easy bi-directional communication between parent and child components.

With that said, VueJS was an obvious next choice for me. Personally, I think VueJS could be used in a lot of situations but I understand that a variety of factors can influence one to choose another framework. So if you have any thoughts about it, I’d love to hear from you. Especially if VueJS hasn’t worked for you in some situation, I’d love to know why. It is good to know some of the bad things others may have experienced with Vue, when all I have had is a positive experience.