'Tech' blog posts

Integrating ARCGIS Online maps for CIS

Construction Information Services (CIS) is the leading supplier of All-Ireland Construction Leads, and it recently launched a new mapping feature for its flagship CIS Online product.

This new feature allows CIS customers to see the exact location of project leads.

The system uses the ARCGIS Online platform, provided by ESRI Ireland, to securely store and display spatial data for each project.

Dovetail had a central role in developing this functionality and was responsible for integrating the ArcGIS platform with CIS Online and backoffice features. The project included:

  • Realtime synchronisation of CIS Project leads with the ArcGIS platform using the ArcGIS Online API
  • Adding secure maps to CIS Online and to backend researcher systems
  • Secure integration of an ArcGIS Online application into CIS Online
  • Using the ArcGIS Online Javascript API to project Irish grid coordinates to longitude and latitude.
  • Backend functionality to assist the research team to verify the location of project leads.

CIS Ireland is one of Dovetail’s most established clients and it celebrates its 45th birthday this year.

"We have trusted Dovetail with our critical systems since 2007. Dovetail’s philosophy is based on “partnership”. They consistently deliver high quality solutions for our business and I have no hesitation in recommending them.”

- Tom Moloney, Managing Director.


Angular, timezones and Datetimepickers

Recently we developed an angular application for one of our Multinational clients with the purpose of helping them to manage accommodation, transfers and flight times. This app had to deal not only with dates but also time and we looked into many different date and time pickers created specifically for angular in order to deal with that. We ended up selecting angular-bootstrap-datetimepicker as our choice.

At first hand everything went fine but during development the daylight saving period started here in Ireland and we noticed the dates being stored in the server started to differ of what we were picking in the app by one hour.

Javascript is somewhat confusing in how it parses dates, times and timezones, even some browsers handle some things in different ways and what was happening was when the datepicker was parsing the date into a javascript object it was also adding local time information e.g. 19/05/2017 11:18:00 +1 hour instead of 19/05/2017 10:18:00, which was actually the “numbers” that were picked in the datepicker. When this information was being sent to the backend, the server was parsing this date object without timezone information so the date being stored in the server was one hour wrong.

We didn’t want the application to be timezone aware, we wanted the users to pick a date on the datepicker and store that date made of the “numbers” that were picked. No smart conversions or anything like that.

The datepicker already provided support to moment.js (which basically the whole world recommends it if you are dealing with dates in javascript) so all what we had to do was to ensure when we are setting dates using the picker the selected date was in UTC timezone.

In order to achieve this behavior we had to add moment-timezone which gave us support to deal with timezones and then changing the code of angular-bootstrap-datetimepicker in the following way:

On the datetimepicker.js file, on the function setTime, one line after case ‘moment’ we added moment.tz.setDefault(“Etc/UTC”) in order to enforce the timezone to UTC every time we are setting a date.

By telling the datepicker to use moment we started to have issues with the angular filter for dates, which simply stop working, so we added another library called angular-moment to deal with that. So the code for the input that shows the date ended being this:

  

Further action: Doing some more testing we discovered that by using moment.utc() we don’t really need add moment-timezone or do the setDefault(“Etc/UTC”) thing. We have not given this a huge amount of testing, but the code would look like this:

 


Dovetailers earn Microsoft Certifications

Congratulations are in order as the following Dovetailers passed their Microsoft Certification exams.

Tomás and Murilo passed 70-461: Querying Microsoft SQL Server

Fabrizio and Kit passed 70-483: Programming in C#

John and Mossy passed 70-532: Developing Microsoft Azure Solutions

Dovetail Values - Clarity, Partnership, Craftsmanship, Commerciality, Progression.

Progression is one of Dovetail's core values and we promote constant learning and improvement. In the fast-moving technical sector, no one can afford to sit still and we are already planning next year's Progression Goals. 

 


Worldpay: remote host closed connection during handshake

Around 4pm yesterday one of our clients began receiving error notifications from Worldpay.  

The message was:

Our systems have detected that your callback has failed.

This callback failure means we were unable to pass information
to your server about the following transaction:

Transaction ID: 1111111111
Cart ID: 1111111111111
Installation ID: 1111111

Error reported: Callback to: https://example.com: failed CAUSED BY Remote host closed connection during handshake
Server Reference: 11111-11-1111:callbackFailureEmail-11111:11111111-11-11

Also, if you usually return a response page for us to display to the Shopper within the time allowed (1 minute), this will not have been displayed.

Googling the error "Remote host closed connection during handshake" shows that the message relates to the requesting service's handling of SSL certificates.

We hadn't changed the client's SSL cert for over a year.  We had not deployed any recent software updates for the client, and we could see that multiple other payment processors, used by this system, were connecting to our server without issue.  There were no errors in our server's Event Log or in the app's Logentries records.

We contacted Worldpay support, who were very helpful.  They told us that SSL certs are cached on their systems, and can be cached for a long time (i.e. over a year).   They also said their systems can't handle SNI.  

So what seems to have happened was that Worldpay's certificate cache was refreshed yesterday around 4pm.  Our client's year-old certificate, which uses SNI, was loaded by Worldpay, and all subsequent connections from Worldpay failed.

Options to fix this include (a) get a new non-SNI certificate and (b) change the callback URL to use HTTP.  

Hopefully this post will assist if someone else experiences this issue.


Custom JavaScript parser vs Jison - Our experience

 

We recently announced QuickDBD, a simple product we made for drawing database diagrams by typing. If you take a look at the QuickDBD app you'll see it converts source code into a diagram. What we needed to make this work was obviously a parser.

After a bit of research on how to approach this problem, we knew that we would have to use either an existing parser generator or build a custom parser ourselves. After narrowing the choices down a bit, PEG.js and Jison emerged as the two most popular JavaScript parser generators at the moment. Out of these two, Jison seemed to have slightly bigger community - a bit more GitHub followers, more StackOverflow questions and a slightly better documentation. It seemed like a better bet so we decided to spend a bit of time playing with it and to try to make it parse the QuickDBD syntax.

We managed to make it parse the first version of our syntax we had a few months back pretty fast. But since the language we came up with for QuickDBD is closer to a data description language than what most people would consider a programming language, we started hitting bumps in the road pretty quickly as well. We soon ended up having to handle multiple edge cases we weren't able to with just Jison and what that meant was overriding Jison behaviour and injecting custom bits of JavaScript into it.

That kind of felt pretty messy so we talked a bit about it and made a decision to go with our own custom JavaScript parser for several reasons:

  • we would have complete control over how the parser works
  • everyone here is very well versed in JS
  • Jison was new to everyone and there is a bit of a learning curve in being able to do stuff with it efficiently
  • it felt as if we were fighting Jison to make it work something it wasn't supposed to more than it felt it was this great tool that was would empower us to do things better and faster
  • a couple of times it was pretty hard to get information on how to do something with Jison so we had to fall back to reading it's source code to figure things out
  • it didn't feel like the right tool for the job

We however did pick up some ideas from trying it out and I believe it made the custom parser we came up with that much better. We wrote a parser that's fairly small, fast and easy to read, expand and fix - which is ultimately what we needed.

I still think Jison is a great tool but it just wasn't a very good fit four for our needs. If you're considering using it, perhaps try it out on a smaller subset of features of your language first and see how you like it before committing to it. You can always go back to writing something custom after you tried it out.

I also recommend you read this very good parser generators vs custom parsers SO thread with pros and cons for both sides.

Hope this helped!


Hello QuickDBD!

Quick Database Diagrams

For the last couple of months we've been working on a side-project here in Dovetail. Martin and Trevor wanted a tool to quickly draw/prototype database diagrams by typing. So, we're happy to announce QuickDBD! We decided to wrap it in a shiny design and make it a little product which we hope others will find useful as well. In time, if there is enough demand we'll expand the feature set. If you have any ideas or suggestions, please let us know on our roadmap Trello board.

In the process of making QuickDBD, a lot of cool, interesting technologies were used and no programming languages were harmed! We used things such as AngularJS, Typescript, JointJS (for diagram rendering - awesome library!), Karma and Jasmine (for testing), Angular Material and SASS on the front-end, .Net WebAPI, xUnit and MS SQL on the back-end and we automated our build-test-deploy pipeline with bower, gulp, TeamCity, Octopus Deploy and Azure. A very interesting journey!

We hope you like QuickDBD same as we do. If you have any feedback, please let us know!


Integrating Karma code coverage with TeamCity

To unit test our Angular apps we use Karma test runner and Jasmine testing framework. Locally we run these tests using a gulp script that takes care of the whole app building process. To ensure nothing is broken before publishing the app to production we run our tests during the continuous integration process using TeamCity.

This post expects you to have a gulp testing process already in place and it won't cover that part. It also expects you to have a working TeamCity setup in place. The post will only help you integrate Karma with TeamCity as an additional build step so you would get something that looks like this in your TeamCity.

Number of passed/failed tests:

The code coverage tab:

There are a few requirements before we can make this work. To help you better understand our setup, here is a sample project structure that we have:

The first thing to do is ensure you have the following npm packages installed and that they are saved in your package.json file:

"karma": "^0.13.22",
"karma-chrome-launcher": "^1.0.1",
"karma-coverage": "^1.1.1",
"karma-jasmine": "^1.0.2",
"karma-phantomjs-launcher": "^1.0.0",
"karma-teamcity-reporter": "^1.0.0",

Next ensure that you have the following set up in your karma.conf.js:

  • "coverage" and "teamcity" in the reporters list
  • "PhantomJS" in your browsers list
  • singleRun set to true
  • our coverageReporter configuration looks like this (this part is pretty important):
coverageReporter: {
  dir: 'coverage',
  reporters: [
    { type: 'html', subdir: 'html' }
  ]
}
  • set the preprocessors configuration to something like this:
'path/to/code/you/want/to/tests/*': ["coverage"]
  • NOTE: we do not have the plugins property set up
  • the rest of options are pretty much standard - add/remove what you need

Now that this is all set up, go to your TeamCity. This is essentially how our client-side build process looks like:

The step that is the main interest of this post is the "Run Karma Tests" step. Here is how we have it set up (create a Command Line step):

This is a slightly modified version of what Karma documentation recommends. The difference is that we are forcing the use of local Karma module and we specify the configuration as a command line param like this:

node node_modules/karma/bin/karma start karma.conf.js

The last piece of the puzzle is setting up the coverage artifact. Go to the General Configuration Settings of your project in TeamCity and add an additional coverage artifact path (the second line):

The important bit (it's simply where our coverage html files are located):

Project.WebApp/coverage/html/** => coverage.zip

Go back and see how we have the coverage/html folder in our project structure. It is set up by coverageReporter karma.conf.js property. This artifact path will take all the files from the coverage/html folder and will compress it into a coverage.zip archive. After the build process finishes, TeamCity will (if it's is able to find the coverage.zip archive inside the artifacts folder) automatically import it as code coverage for the project and you will be able to navigate to the "Code Coverage" tab for that specific build. If you have any tests that don't pass, this will also fail the whole step, stop the build and prevent it from ending up in production.

Hope this helps. Cheers! :)


Visual studio 2015 real-time CSS editing

I was working on some updates to MenuCal this week. This morning while completing some CSS styling on a new form, I discovered that the CSS was being updated in real-time in Chrome as I made changes in visual studio. This is a huge improvement to my workflow, as I like to style and preview as I go. I was able to drag my CSS editor over to another screen and work away while the styles in Chrome updated instantly. No more hitting save, and refreshing the browser. Thank you Visual studio 2015!! I've seen other tools do this for quite some time, but it's nice to see it in the IDE I use every day.

But let's not get too excited. I mean who uses plain old CSS anymore? We've been using SASS on new projects and unfortunately this lovely little feature is not present out of the box for SASS. I will take a look around at VS plugins that might do that, and report back if I find some elegant solutions.

Update 2nd August, 2016: I tested out one of our projects with SCSS and Sassy Studio. While it's not as elegant as the live CSS preview, it does detect the CSS changes after they are compiled, and the browser updates the CSS.


This week in tech

I'm jotting down some notable tech news we've been discussing internally (in our slack #techtalk channel) this week.

We use New Relic on a number of applications, it's a great tool for highlighting performance issues in applications. Microsoft has always been somewhat in that game, but their new offering built into Azure is called "Application Insights". It looks to be a direct competitor to New Relic. It also has logging and a query engine to go with it, so it may also be aiming for cloud logging providers too (like Log Entries). https://azure.microsoft.com/en-us/documentation/articles/app-insights-overview/.

Trevor uses a mac (boo!), and we're a Microsoft development house. At times he struggles to find the right tools to work in a primarily windows environment, and he usually resorts to a virtual machine or RDP. We recently found this tool called Wagon (https://www.wagonhq.com/), and Trevor has been using it and enjoying it. Wagon is built on Electron, another tool we have been keeping an eye on lately. Fabrizio is especially enamored by it.

Apparently we care about API versioning. I'm not sure, but other people care about it more than me: Your API versioning is wrong, which is why I decided to do it 3 different wrong ways.

VHS won! But only barely. https://www.theguardian.com/technology/2015/nov/10/betamax-dead-long-live-vhs-sony-end-produtionhttp://news.sky.com/story/remember-vcrs-production-to-end-as-sales-slump-10509632.

Javascript jokes are so hot right now:

Lastly, John found this. Have we gone too far?


Team City - Update Packages

Here at Dovetail we love Team City and Visual Studio.

We recently updated our Team City configuration to allow projects to be built using Visual Studio 2015, C# 6, and to use the latest Nuget package manager.

In doing so, we discovered a very peculiar setting deep within Team City that caused one of our projects to break on build and break once deployed.

The Build Failures

After updating, we ran our build and the compiler threw an error saying that it could not find a specific version of a Nuget package. For example, our packages.config within Visual Studio specified we use Nuget to install Newtonsoft.Json version 7. However Team City reported that the project needed Newtonsoft.Json version 8.

We made the decision to update all affected nuget packages to the latest versions, pushed our project and Team City built it successfully.

The Deploy Failure

We then ran into our next problem. The project was deployed but there was nothing on the screen. We opened up Chrome developer tools and found that JQuery was missing. This is a project that uses a lot of JavaScript files and it built and deployed with no problems before.

Looking back at our Octopus Deploy package we found that the JQuery file we were referencing and pushing to our repository was not there any more. However, we did see the latest version of the JQuery min file. Our file was being removed and replaced with the latest JQuery min version.

The Update Package Setting

We soon found the setting buried deep inside the Team City "build steps" screens:

 undefined

Within the NuGet Installer build step is a setting which, when turned on, updates all your packages. This sounds great in theory but when you run into build and deploy issues this will cause headaches.

undefined

 

The text underneath states "Uses the NuGet update command to update all packages under solution. Package versions and constraints are taken from packages.config files". Whether this is a bug in Team City or not, this text seems very vague for an "Update Packages" function.

Be careful, because when checking this check-box, Team City will not read the packages.config version numbers and instead it will download the latest version of every package.

Update: Team City have been back to us and they're going to update the explanatory text on this checkbox to make it more clear.