Static sites and Functions

This is the fifth and last part of a series of posts I am writing about building a static site with VueJS. In this post, I will walk-through how you could use Functions-as-a-Service for your next project…or your current too.

Static sites typically don’t get all of the infrastructure attention, that other stuff does. Many developers still think that SPAs, whether static sites or not, need to be hosted with an always-running server infrastructure. It is unnecessary. With the advent of ServiceWorkers and the Progressive Web App movement, you really don’t need a server running all of the time.

Most devs are familiar with using a CMS like WordPress and then buying a domain to serve a website. Most websites don’t need all of that infrastructure. The price is a modest $4/mo according to their pricing page, you just get the basic with the paid plan. Not a big deal, though. But if you wanted to do SEO, custom analytics etc., you are looking at the next pricing tier or perhaps the most expensive one, at $25/mo, if you are looking for a few more knobs/levers to turn.

This is the architecture I used for Net Your Problem.

Static websites architecture with Functions-Page-1
Fig.1 – A simple cloud architecture for SPAs.

I have automated the part where I build the VueJS app and upload it to the Azure Storage account using a PowerShell script (see this gist), which is purely based on the AzureRM PS module.

Great. Now, let’s talk about how these infrastructure systems talk to each other, to cohesively power your next project.

The inner details

A CDN works by aggressively caching static resources (JS, CSS, HTML, PNGs etc.). You can also give a hint to a CDN service to cache additional mime-types by setting the Cache-Control header attribute. A typical CDN service has “edge” nodes all over the world. When you provision a CDN service from any of the cloud providers, you are not choosing the edge nodes in a particular region. CDN services are globally distributed, by default. Each CDN service does their billing differently. For example, I know that the Azure CDN service offers a tiered pricing model based on a group of countries in each tier. So traffic from different countries will be billed at different rates, based on the amount of data transferred from the CDN to the clients (browsers).

As shown in fig.1, the CDN is connect to the Function App, meaning that the CDN will source the static assets from the Function App. But the Function App in turn is connected to a storage account. Technically, all of these can be services from any of the 3 major cloud providers (Azure, AWS, GCP). It doesn’t make sense, though, to create individual services in multiple clouds. You would be charged more for inter-data center data transfer. So it is best to co-locate all of these, except of course, the CDN, which is always global, regardless of whose CDN service you end up using.

The connection between the CDN and the Function App is pretty simple, as it is just a matter of specifying the origin settings for the CDN. The connection between the Function App and the Storage Account requires a little bit more than just specifying the URL. We have to detect the incoming request at the Function and proxy it to the storage account, to let the storage account serve the static asset. Essentially, the Function App serves as a reverse-proxy here for some URL patterns, and for others, as a service that may or may not return a response, i.e., if there is an actual function that can handle the request.

Bonus! Let’s talk about automation

Let’s introduce a CI/CD pipeline into the mix. Azure has a really good all-in-one Ops platform called Azure DevOps, previously known as Visual Studio Team Services or VSTS, and even before that, Visual Studio Online. Anyway, the point of the platform is like Bitbucket or GitHub, where you have everything in one place. CI/CD pipeline integrations, release management, project planning (Kanban as well as sprint-based, whichever you are into), private/public repos, wikis, and also a private feed for your Nuget (.Net packages), npm or maven packages too!

Don’t take my word for it, though. After all, I am just some random programmer on the internet. People say a lot of things. But seriously, you should check it out.

Static websites architecture with Functions-Page-2

Here’s the screenshot of the pipelines page showing you the CI and PROD build pipelines for Net Your Problem.

Screenshot_2018-10-09 Builds - Pipelines

Here’s the CI/CD pipeline in Azure DevOps.

The pipeline is simple. Run npm ci –> npm install –> npm run build –> upload to Az Storage –> store artifacts. That’s it. I have a trigger for this pipeline, which would kick-off this build every time a branch is updated through a merge.

Admittedly, all of this may look like overkill. Trust me, it is not. I spend about a minute to run the build and then to upload the files to the Azure Storage. Then, sometimes, I have to purge the CDN cache, because, well, it works too well sometimes :). Overall, I could spend anywhere between 1-10 mins, and on average ~5 mins, deploying some changes. Now, repeat this several times as I am actively developing something or want to see how things look in Dev, or I want to show something to my ahem client (my girlfriend), the time investment adds-up really quickly. This setup allows me to focus just on the coding part and push up my changes for a quick review on the dev site, before I create a release for the PROD and have approved changes go to the live site immediately. All of this is a pleasant experience for me. To my girlfriend, it makes no difference, and that’s a good thing. She just sees the dev site and the live site. That’s it.

You see, the delays in development, often affects our customers. When I have a pipeline, that works well, my customer doesn’t get affected by it. They just see what they need to see. To them, in the end, it matters if they are able to see what they want and if it works. If the process gets in their way, they simply won’t get it. If I could use the overused automobile world for an analogy. This experience would be akin to taking our car to a shop for an oil change. At the end of it, we just want to drive our car out of the shop with a new oil filter, and fresh oil. We don’t care and most of us don’t want to know how they were able to do an oil change without an appointment. On the other hand, if the oil change took too long, then we want an explanation and all of the shop’s fancy equipment and ISO certifications wouldn’t save them from our negative experience.

Advertisements

Using machine learning to classify NSFW images

Hi, all! I have a favor to ask. I have an alpha version of my app, Transitions, that can classify images as NSFW with an 80% or higher accuracy, most of the times (top-1 accuracy as it is known in the ML world). It uses Mobilenet V1. The predictions are made on-device. So your data is not used for predicting whether an image is NSFW or not.

I was wondering if any of you would be interested in signing-up to be in an Internal Test Track? I already run an Open Beta program but before releasing a beta, I’d like to test it with a closed loop of users. Sorry, but my app is Android-only. You’ll need a device with Android 6.0 or higher. So please PM me if you are interested. Once you are part of the testing program, I am hoping you would also be OK to submit feedback so I can improve the model’s accuracy.

Why NSFW classification?

Transitions serves user-generated content from flickr, 500px, and Unsplash. Even with category filtering, inappropriate images show up all the time. I currently have an ESRB rating of Teen on Google Play, because of this. I can’t change that rating;¬† even with the automated classification and dynamic blurring of such images. But I want to give my users the ability to browse high-quality photos on Transitions without the uncertainty of what they’ll see next.

What do you (as a tester) get from this?

Uh, the joy of helping me? Early access? Take your pick.. ūüėÄ

Remote debugging an embedded WebView in Android

And there I was, thinking that I was done with AzureStorageExplorer for Android v1.0.0. But, nope. In my last round of testing, I couldn’t even login using my work account. The Sign In button didn’t do anything. I went back a few versions to see if the Android update for WebView¬†could have done something to break it. That wasn’t it. The OAuth client library I was using had the recommended code for enabling Javascript in the WebView client so I knew that wasn’t it. I was able to login through the Azure OAuth flow using my personal Azure account. So, I knew there was something about the enterprise account login screen, which could have changed recently that ended up breaking the authentication in the app. It turns out that it did. Read on to know how I found out.

The first step was to prove that something was indeed broken in the login screen for enterprise accounts in Azure in embedded WebViews. To do this, I looked up Chrome’s remote debugging options and found this page describing how to remotely debug WebViews in mobile apps. So, I enabled the setting as recommended by Google, which required adding this piece of code in my activity’s onCreate(Bundle) method.

if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
  WebView.setWebContentsDebuggingEnabled(true);
}

I ran the app, launched Chrome’s Developer Tools, and¬†connected to the app’s WebView. This is what I found..

2016-12-26_09-49-21

That last error in the console is what was logged when I pressed the Sign In button after entering my work account credentials for Azure. It turns out that the page’s JS now has code to store something in the browser’s localStorage. The reason this isn’t a problem, if you were logging into Azure through a desktop or even a mobile browser, is because localStorage is enabled for every site by default in the non-embedded browser view. But for a WebView in an Android app you need to explicitly enable the DOM storage, just like you need to enable JS through the WebSettings class.

After finding this, it was easy to know where I needed to make the fix. So, I set out to fork the source repo of the OAuth library I was using, made the necessary changes and created a pull request. If you are using the same library and have run into this issue, you could clone my repo and build the project to produce the patched .jar (or an .aar), which you can use in your project directly until the author of the library can get to my pull request (if at all!).

Unofficial Tesla Android client for controlling your Model S

Tesla Motors
Tesla Motors

GitHub link: https://github.com/praneetloke/MyTesla

But there’s already an official app from Tesla for the iOS and Android, then why this? Because I wanted to and besides there aren’t any open source Android clients. There are couple of Ruby clients and a node.js client. The node.js client is particularly of interest to me since it has visual examples of each API it supports. The telemetry streaming API is my favorite. If you remember, that’s what Tesla used to debunk the NY Times reporter for his fake report on the Model S sometime ago.

Anyway, back to the MyTesla client. You can fork it, download it, modify it, do whatever you want with it. This client is unstable, unofficial, and most importantly, unverified (since I don’t own a Tesla Model S, unfortunately). So please use this with caution. If you find bugs, please raise an issue in my repo or you can fork-fix-pull. If you are able to help with its stabilization by testing this on your Tesla Model S, please let me know. You can hit me up on G+ or LinkedIn.

There are 3 REST API clients for your use in this project. LoginClient, VehicleStatusClient and VehicleCommandClient. I have already pulled in these 3 REST interfaces into a custom Android Application class called TeslaApplication. I did this to remove any dependency from consumers having to pull in AndroidAnnotations as well. On a side note, you should check out AndroidAnnotations. It is awesome! Anyway, in your project, you can simply extend the TeslaApplication and be on your way. You don’t need to put anything in it. I have not actually tested to see if such an extension is actually required if you were to reference this project but be my guest.

I have also made a LoginActivity which has a boilerplate login form which you can present to your users. It handles submitting the email and password to the API and inspecting the response to see if it was successful. I actually plan to change this to a login dialog instead or perhaps have both since a dialog only needs a layout. Then you can choose either depending on your needs. When login fails, it currently doesn’t do anything. I am yet to work on that. I also need to work on some cookie transfer from LoginClient to the other two clients because from what I saw in the AndroidAnnotations sources, there doesn’t seem to be a unified storage for cookies acquired by REST interfaces.

When you look at the library I made, you will notice that I didn’t use primitives. That’s because of the nature of the API itself. It’s unofficial and there are unconfirmed properties whose values are unknown and sometimes null. So this being Java, I couldn’t use primitives in some cases. For those that had confirmed values I could have used primitives but I felt I needed to be consistent rather than have you guess what you’ll be using when you inspect an object. And yes, I am talking about primitives and objects because I actually went ahead and created model classes (POJOs) for all of the endpoints. This should make interaction way easier. It uses Gson for type conversion. I chose Gson over Jackson mapper for its lightweight and performance. Gson doesn’t have all of the features Jackson has but it does the job, fast too.

If you have watched this clip of the guy issuing commands to a Tesla Model S, you’ll be at least half as excited as I was to find a REST client and play with it if you have a Tesla Model S. Of course, you would more likely already have the official Tesla app. But if you are into programming and diving into things on your own, this is for you. I wish my VW CC was capable of something like this.

Credits

  • Tim Dorr (and many others that commented on each API endpoint in the Apiary blueprint) for his excellent Apiary documentation based on his findings. He has implemented his very own ruby implementation of the API here.
  • AndroidAnnotations
  • Spring Android

Other clients

  • node.js
  • Ruby. For Ruby, there’s also the one from Tim Dorr himself.

My struggles with Android’s CalendarContract introduced in ICS

While working on some upgrades for Squiggly I wanted to subscribe to Android’s calendar provider events. This is how you would do it.

Assuming you have a receiver in your app, if you have declared it in the manifest xml, then add this to get provider_changed intents. Thanks to this stackoverflow question.

<intent-filter>
    <action android:name="android.intent.action.PROVIDER_CHANGED"/>
    <data android:scheme="content"/>
    <data android:host="com.android.calendar"/>
</intent-filter>

Now, the above intent-filter is missing a few things that could have made my interaction with the calendar provider easier. They are the attributes paths, pathPattern and pathPrefix for the “ part of the intent-filter. Unfortunately, even specifying those attributes won’t do you any good because the CalendarProvider class itself doesn’t specify those parameters on the intent that is fired from within.

The other problem is that the intent that is fired from CalendarProvider doesn’t even carry the event id that is passed to it from the methods that call it. See sendUpdateNotification from CalendarProvider2.java in the JellyBean source code here. This method sends a message to its Handler instance which then broadcasts the intent for provider_changed action. ref: this line. The problem is this simply obtains a message from the message pool and passes the value that sets the what of the Message instance returned. Instead, they could have done this:

//get a message using the obtainMessage(int what, Object obj) version
Message msg = mBroadcastHandler.obtainMessage(UPDATE_BROADCAST_MSG, new Long(eventId));

With the eventId now getting lost with the aforementioned line#4478 from CalendarProvider2.java, the handler now doesn’t even add any extras to the intent before broadcasting it. The control eventually reaches doSendUpdateNotification, which as you can see does nothing but set the action in the intent and broadcasts it.

What’s even more frustrating is that if you look at the usages of this method, the callers actually do pass the eventId and the actions that trigger a call to this method are actually the ones I want to know the event ID for!!

Usages for sendUpdateNotification in JellyBean r1
Usages for sendUpdateNotification in JellyBean r1

Well, looks like the latest KitKat revision still hasn’t changed this. Until then, I guess I will have to scan the instances table whenever the provider_changed action is received. ūüė¶

If you care about this issue and want to bring this to the Android team’s attention then please vote for this issue.

Dynamic entries in Android’s MultiSelectListPreference

So I was finally excited that my brother and I decided to drop support for Gingerbread for the next iteration of Squiggly and that allows me to do things like use the MultiSelectListPreference and use the local Calendar API exclusively.

But there is a drawback with MultiSelectListPreference. You have to pre-define the string arrays for entries and entryValues. This sucks. This is why I have now added CustomMultiSelectListPreference to CustomPreferences. I have followed the same pattern which I did with CustomListPreference by implementing the IDynamicProvider which I introduced a while back when I needed to populate a regular list preference dynamically.

This is the class to reference in your preference xml file.

To use it:

<com.myappfactory.preferences.CustomMultiSelectListPreference
            android:key="@string/calendarsToMonitorKey"
            android:dependency="@string/enableCalendarBackgroundServicesKey"
            android:title="@string/calendarsToMonitorTitle"
            android:dialogTitle="@string/calendarsToMonitorDialogTitle"
            android:summary="@string/calendarsToMonitorSummary"
            customPreference:dynamicEntriesProvider="your.app.package.ClassThatImplementsIDynamicProvider"
            customPreference:dynamicEntryValuesProvider="your.app.package.ClassThatImplementsIDynamicProvider"
            customPreference:selectAllValuesByDefault="true"
            android:entries="@array/empty_array"
            android:entryValues="@array/empty_array"
            android:defaultValue="@array/empty_array"/>

selectAllValuesByDefault does exactly what it says. It selects all the values that are dynamically added. The class that provides these dynamic entries and entryValues need to implement IDynamicProvider. The no-argument version of populate method will no longer be supported. So be sure to implement the populate that accepts a Context. CustomMultiSelectListPreference first calls populate then calls getItem. I have tried to keep the same naming convention that Android’s adapter classes use. So you will see getCount and getItems in there.

Here’s a preview of what it will look like:

CustomMultiSelectListPreference
CustomMultiSelectListPreference

Ford TDK

Guess what?? Last week, I got the Ford TDK which I won from participating in an app idea contest held by Ford. Here’re the pictures!

TDK power cord and instructions manual.
Instructions and power cord for the Ford/Lincoln TDK 3.0
Ford/Lincoln TDK
Ford/Lincoln TDK

[UPDATE] Weird problem with GridView OnScrollListener and list navigation listener

It turns out that the problem was much simpler than I had originally thought.

The Problem

It was actually two fold. The scroll listener for some reason fires the change event as soon as the gridview’s adapter is set even when there are no items in it. My scroll listener was something like this:

private AbsListView.OnScrollListener mOnScrollListener = new AbsListView.OnScrollListener({
  @Override
  public void onScrollStateChanged (AbsListView view, int scrollState) {}

  @Override
  public void onScroll (AbsListView view, int firstVisibleItem, int visibleItemCount, int totalItemCount) {
    if ((firstVisibleItem + lastVisibleItem) >= totalItemCount && !mAsyncTasksPending) {
      loadMore();
    }
  }
});

The above code ended up calling the loadMore(); even before the gridview’s adapter was loaded with the first page results from the service. That ended up making two requests wastefully. The second problem is because of how I had declared the loadMore(); method. I use AndroidAnnotations to reduce boilerplate code. I annotated loadMore(); with @Background to execute the service request in a background thread to prevent the UI from locking up. But, I didn’t need to do this since I was already taking care that the service request be executed in a background thread using another library called async-http, which by the way is freaking awesome! My guess is, when I set the method to be executed in a background thread it was holding up the gridview’s listener thread somehow which caused the events to accumulate in the call stack. This would probably explain why when I paused the activity and resumed it, it would function normally and the moment I scrolled it and caused it to call loadMore() it would freeze again and none of the callbacks would fire.

A little more than the solution

First, what I ended up doing was removing the redundant @Background annotation from loadMore();. Then, I realized that my activity was a little bloated. So, I created a fragment and shoved the gridview, and managing its adapter into the fragment. I was then able to keep my activity lean. By doing so, I have also opened up the possibility of introducing additional fragments to handle different views if necessary in the future which makes this all the more easy to manage without bloating my activity.

Next, I updated the gridview’s onScroll callback. It was still being called as soon as the adapter was set in the gridview. Initially, the adapter is empty and I don’t want it to call loadMore(); when the first page results haven’t been filled in the adapter yet. Loading the first page is handled through the list navigation listener since that sets the currently selected item for which I then go and fetch the list of images. So now, this is how my updated OnScrollListener looks.

private AbsListView.OnScrollListener mOnScrollListener = new AbsListView.OnScrollListener({
  @Override
  public void onScrollStateChanged (AbsListView view, int scrollState) {}

  @Override
  public void onScroll (AbsListView view, int firstVisibleItem, int visibleItemCount, int totalItemCount) {
    //only load more if the visibleItemCount is > 0, and the user has reached the last row of visible items
    if (visibleItemCount > 0 && ((firstVisibleItem + lastVisibleItem) >= totalItemCount) && !mAsyncTasksPending) {
      loadMore();
    }
  }
});

Why I dumped my custom image manipulation code from Transitions

After spending several nights and weekends implementing and testing the code I had written for image resizing, it still didn’t work for some popular devices. This was unacceptable. As I gave this a lot of thought, I realized that I was doing it wrong. No. Not just the code. My effort. As much as the engineer that I am, I wanted to make it work. I was very interested in solving it. I wanted to own it much like every other coding problem I try to solve. But, as an Engineer, I also don’t want to “reinvent the wheel” if I am not going to improve it. I certainly wasn’t improving it.

So, for¬†the upcoming release for Transitions, I trashed all of the code I wrote for image manipulation and simply popped in Square’s Picasso library. Why? Because, it’s simply that much less code I need to change/maintain. It is so much easier to look at my code now with all the complex bitmap decoding stuff gone. I’ll let the experts handle that. And, now I can concentrate what really matters. Putting features together. I am still innovating with Transitions. I added a feature to my open source library CustomPreferences which allows you to load a list preference from the database dynamically. It’s very useful when you let your users build their own database of things and you want to allow them to pick from that list. Currently, you can’t do this with the preference classes out-of-the-box. But, you can, with CustomPreferences.

 

 

Closure function vs defined (named) function

I wrote a performance test on jsperf.com to measure the difference between calling a closure function from within a loop and calling a defined function within another loop for the same number of iterations. See the difference and run the test yourself.