Nigel Meakins

Nigel Meakins' Blog

Google Cloud Spanner

This is the first of a series of posts around the Google Cloud Next 2017 conference and some of the services showcased there.

What is it?

This is a new Managed Relational Service with two big considerations:

  • Horizontally scalable
  • Transactional Consistency across globally distributed database nodes

Scaling is apparently effortless, with sharding just happening as you’d want it no questions asked, no effort required. It supports ANSI SQL 2011, (with some minor changes required around data distribution and primary keys), uses relational schemas, giving tables and views that we are all familiar with, and will be auto-replicated globally (this feature coming later this year). It is going to GA on May 16th 2017. It will support databases of 100s of TB in size, with 99.999% (“five nines”) availability and offers single digit millisecond latency.

So why would we need this? These considerations are of significance in applications that require precision on event times to remove any potential for transactional indecision that would otherwise be possible due to a lack of temporal granularity, whilst being available across the globe at scale. Imagine for example two bank account holders withdraw funds at the same time, one of which will take the account into the red. Who should get declined, holder A, holder B or both? Although not critical many types of database applications, it is essential in Inventory Management, Supply Chain and Financial Services for example, where sub-second precision really is essential when there may be hundreds or thousands of transactions every second. The demonstration given at Google Cloud Next was one of a ticket ordering app receiving around 500 thousand requests for event tickets across the globe (simulated by a bot network of pseudo event-going customers). You can see the YouTube video of this here. Apparently this combination of true transactional consistency and horizontal scale is simply not possible with other sharding approaches.


Hybrid Transactional Analytical Processing (HTAP)

This combination of Horizontal Scale and Transactional Consistency allow for what Gartner have termed “Hybrid Transactional Analytical Processing” (HTAP) applications, where massive analytical capabilities are possible from within the application. So Operational Analytics at really big scale, no data movement, no additional platforms, just straight out of the Operational system. That is really quite a big deal. You can read more on the benefits of HTAP in this free Gartner document here so I won’t elaborate further. Suffice to say not a sliver bullet for all analytical needs across a complex ecosystem of applications, but definitely a game changer when there is only one instance of an application to report from. And if your database can scale to handle it, maybe you will be putting multiple systems into the same repository.


Eh? Is that a typo? Nope. That’s right, another term to get your head round, just when you thought all the NoSQL stuff was settling in. Okay, not really, just a consideration for what tries to be the best of both worlds, at least that’s the idea. So, for the two main considerations for the database world, scalability and consistency, NoSQL offers massive scaling at the expense of transactional consistency, SQL offers transactional consistency over scaling, ’NewSQL’ claims to offer both. Simples right?

The CAP Theorem - Two Out Of Three Is The Best You Can Hope For

Proposed by Eric Brewer in 1998, CAP Theorem is a big consideration when we talk about distributed database systems. You can read about this on Wikipedia here.

In essence It states that Consistency of transactions, Availability of the system and the tolerance of network Partitioning, where messages may be lost over the network, is not possible in unison. You can have two of these but never all three. According to Eric Brewer (now working at Google but not on the Spanner team) Google Cloud Spanner doesn’t overcome this, but it comes pretty close. I won’t go into the innards of CAP here, but suffice to say you  need to consider the compromise that you are making with the missing C, A, or P element and how that will impact your system. Google describes the system as CP with very good A. Five nines A on this sort of scale is pretty good I’m sure you’ll agree.

So Two and Three Quarters Then? Okay, Tell Me More

So what’s in the secret sauce that apparently gets us close to all three of our CAP Theorem wishes? There are three main ingredients that make this happen.

  • Google “TrueTime” time synchronisation, which is based on atomic clock signals and GPS data broadcast throughout the Google network backbone, to ensure transactional consistency. Each datum is assigned a globally consistent timestamp and globally consistent reads are possible without locking. TrueTime defines a known time that has passed, and a time that definately hasn’t passed yet, so a range of time, albeit generally very small. If this range is large for some reason, Google Spanner waits till it has passed to ensure time consistency across the system. There is a lot more to it than that, which you can find out from Eric Brewer here, so I’ll leave that to the experts to explain.
  • Consensus on writes across nodes is achieved using the Paxos algorithm, (nothing to do with Greek 18-30 holidays or Sage and onion stuffing), a robust approach to ensuring that writes are consistent over available nodes, using a Quorum approach similar to the ideas round in high availability fail over. So the system determines what is an acceptable level of availability of nodes to guarantee writes are safe.
  • Dedicated internal network within Google data centres that allows optimised network communications across nodes globally distributed within Google’s various data centres.

Summing Up

This technology has been used internally for 10+ years, originating from a need to use a better option than the standard MySQL sharding that was the previous approach at Google when needing relational scale-out. It is the underlying technology for the Google F1 database that is used for AdWords and Google Play, both critical for Google’s success and having some demanding requirements for both scale and consistency. You can read more on Google F1 and the internal Spanner database project here.For the Google Spanner product blurb, see here. You’ll see that this is not bleeding edge risky tech. The term ‘battle-tested’ was used far too often for my pacifist side but probably hits the nail on the head (preferably with a foam hammer of course). Ten years running some of the most demanding data processing needs on the planet probably means you’re in safe hands I guess. The potential for HTAP and a truly global relational database that scales effortlessly is going to cause some real changes. Of course Microsoft have not been resting on their laurels in this area either and have this week announce their competitor offering Cosmos DB, although that is essentially a NoSQL approach with less of a consistency focus. More on that in an upcoming post…

Google Cloud Next

Google Next LogoI had the fantastic opportunity to attend the Google Cloud Next conference at the ExCel centre in London last week. There were some really amazing things on show there as you’d expect and one or two announcements that were of note to the Data and Analytics world that we at Adatis move within. I’ll be sharing some of these over the next few weeks in a series of posts. First up, Cloud Spanner.

MS Teams–Teams or Channels?

When planning your MS Teams rollout and how to structure your teams there are some simple points to bear in mind.

Teams is for Teams…

As the name suggests, MS Teams is about, well, teams really. Thanks for that fantastic nugget of genius Captain Stating-The-Obvious, what would we do without you? Therefore if you have teams of people already organised for projects, then creating an MS Teams team for each of these makes logical sense. Should you have information that is only for this team, it can be contained within the privacy of the team, as team boundaries are the only point at which access can be enforced. There are no private channels (although this may be something coming down the release pipe though no real indication of as and when).

Teams is Flat

You may not realise this at first, but there are no real hierarchies within MS Teams other than Teams being the parents of Channels, which in my book doesn’t really count. So, no ‘Sub-Teams’. And no ‘Sub-Channels’ either. You will need to think about naming conventions to enable some logical sorting of your teams if you are to avoid a jumbled mess on the UI. Channels within Teams can be collapsed within the Desktop App, but not within the mobile apps. To make mobile users’ lives a little more tedious, the search functionality does not extend to actual team and channel names, so you’ll be whipping your finger frantically up and down the screen to find that team or channel you are interested in. You can ‘favourite’ channels but not teams to help with this, but there is still lots of room for improvement in this area of the application. By organising into teams at least you can manage membership and people will only see those teams and channels they are members of, in addition to any channels they have explicitly chosen to ‘follow’.

Teams and Channels are Hard to Move

Bear in mind that once created, it is not easy to move chat and other team content that is stored outside of SharePoint Online to another team. Don’t assume you can fix things up as your (flat) ‘topology’ develops.This is not something you really want to have to try, and even if you did your options for relocating are limited as things are, well, flat. If you are thinking of having a lot of channels within a team and then reorganising into other teams when the number of channels starts to become overwhelming or a better structure presents itself, this isn’t really going to be easy. Microsoft have not really provided any provision for reorganising your content. The assumption appears to be that users know what their teams are going to look like and what these teams will contain, and that isn’t likely to change, so why add this functionality. I’m sure as the adoption of the service picks up Microsoft will realise that this isn’t really going to cut it in large organisations, where restructuring is sometimes a quarterly activity (“What do you mean we’re moving desks again? My bonsai is only just getting acclimatised!”). For now however best to exercise some forethought and try and avoid this painful exercise. A well thought out demarcation for your teams will avoid a whole world of pain in the not too distant future.

Below is a simple suggestion for how you may start to structure teams for client projects and R&D within a medium sized organisation.

Company TeamsYou may want to have a team that is purely for ‘global’ client stuff outside of projects, in addition to the Project-focused teams.
Client Teams and ChannelsWithin each team the various channels allow the team members to structure their communications in a focussed manner, for the attention of the members of that team only. As you can see, once you start segregating your topics into channels, these project teams are going to be getting rather busy on the channel listing, as this hypothetical project alleviates to. Bunch all of the Client 1 channels into a single team and you can see how things will soon become difficult to manage.

R and D Teams and Channels

The same thinking applies of course for non-client work structuring, such as for our R&D endeavours.

You could of course have opted for bundling everything for Client 1 into its own team, or likewise for R & D, with an increased list of channels therein. With each additional project you will have more channels for your larger audience within the team (as they now contain members for and increased remit) to have to wade through, which over time is probably going to become unmanageable unless you are only looking at a small number of projects. This way there may well be monsters, or at least swamps, quicksand and other things you’ll probably wish you’d avoided…

Microsoft Teams

At Adatis we have been investigating using Microsoft Teams for collaboration and communication within client projects and across the Company. When working on project teams, big or small, these aspects of daily work can really have an impact on productivity and that sense of being part of the team, especially when some team members are not always co-located. With MS Teams only recently going to GA, we have also used Slack for similar uses cases, which, being the main competitor in this space has a lot to offer. I won’t go into too much detail regarding the functionality of each product as there are plenty of articles relating to this already. You can find details regarding the two products at the links below:

So What is MS Teams?

MS Teams is a service available through Office 365, free of charge. It leverages various existing Office 365 products to assist with document collaboration, chat, meetings, conference calls etc. as required within a team. Team membership is however limited to those people within your Office 365 domain, which does mean that those teams requiring a mix of internal and external resources may not find the service suitable for their needs.

Teams and Channels

Teams can be public or private. Public teams are available for anyone within your domain to join and leave as they see fit, whereas private teams have their membership managed by a team owner. All content within the team can be seen by all members, except for one-to-one conversations.

Each team hosts one or more channels which are essentially containers for conversations and documents regarding the various subjects of interest to the team. Members can follow channels that they are interested in, receiving notifications when items are posted on the channels. All this is pretty standard stuff, and something that Slack does very well. The main strength of MS Teams is in the use of the Office 365 services for collaboration. Each team has a SharePoint Online site with all the usual functionality, and each channel has a top level document folder within the document library for hosting the various documents used by the team. Further integration with OneDrive allows rapid sharing of information within the team and a robust document store to be created, all from within the MS Teams application. Each channel has a number of tabs that can be used for browsing a variety of Office 365 or external website content. Documents of interest can be edited within a tab, meaning that collaborating on a document is as easy as if using SharePoint Online (which is exactly what you are doing of course).

Using MS Teams

In order to benefit from the full functionality offered by MS Teams it is best to download the MS Teams Desktop application, which is available to download for Windows 32bit/64 bit, and Mac OSX 10.10+ here. The application is also available for mobile (Android, Windows Phone and iOS) at the usual online app stores, and a web version of the app is also available from the Office 365 landing page, which offers the same functionality as the Desktop app. Mobile clients do offer a reduced level of functionality, missing some of the additional Office 365 service integration and administration functionality such as for creating new teams and channels and managing membership. If, however, keeping abreast of chat and limited editing of documents (using the official Office 365 mobile apps only) are all you need on the go, then the mobile app should suffice for most people until back at your desk.

Having this all available in a single place together with ongoing conversations with colleagues using the chat functionality does give a more focused approach to working. All this does promise to streamline all that project activity that can sometimes get rather disjointed and muddled as a result of the various methods that may be employed for communicating on projects, such as email, SharePoint/CMS Skype For Business chat etc.

So How Does it Really Measure Up?

MS Teams looks very promising for all those day to day tasks relating to collaboration and communication within a project team. It is still however a rather v0.9 product in some areas however, with some key functionality still going wanting. There are various comparisons made between MS Teams and Slack which is the current front runner in this space. Slack is a more mature product that is generally adopted for teams that do not subscribe to the Office 365 platform, such as those within the Open Source community and those who do not to use Microsoft products for development.

Key areas for improvement are the lack of real integration with TFS, (although VSTS is slightly better catered for) and lack of private channels within teams, although the latter is understood to be on the list for H1 2017. In my view the main points of interest for comparison can be summarised below:

  • MS Teams allows easy integration with SharePoint content, OneDrive and Office365 domain membership. For Microsoft Office 365-based organisations this will be a big advantage for collaboration needs.
  • Slack provides External account access for those members outside of the domain. This is presumably possible due to the lack of reliance on Office365 group membership and less integration with the underlying infrastructure (SharePoint Online etc.)
  • Slack provides private channels within teams. This removes the need to create a separate team to provide a private communications channel.
  • MS Teams is free to all Office 365 users.
  • Slack requires a chargeable plan for functionality such as conference calling that is available for free within MS Teams.
  • Slack provides more external application connectivity and integration than MS Teams.
  • Slack provides additional authentication providers such as OAuth via Google.

In my view the centralised nature of document collaboration and communication make MS Teams really worth considering. Here at Adatis we look forward to using this service across new projects to further improve on our Agile approach to BI delivery.