Interesting one today – standing up a Cosmos DB to record the output of a CycleCloud job run which happened to be written in C++ and started getting “Failed to read item”. Data Explorer stopped showing the results from the item when browsing.
Issue was that our new id had been delimited with slashes and Cosmos DB didn’t like it. If you get “Failed to read item” when clicking through then you might have a character in your document Id that Cosmos doesn’t like.
There are some awesome folks out there who share their hard efforts so the rest of us can have an easier job. A few of these that have been really useful sit around work against the REST APIs of key Azure services.
My days of day in day out development are over so I find a lot of my automation “glue” mashing up deployments relies on PowerShell with the odd bit of CLI. Most is a little bit of scaffolding to deploy ARM templates but occasionally a requirement to work with the data plane of a resource appears and I have to resort to manual config.
ARM Template support for configuring resources is always improving but due to timing this isn’t always possible. Sometimes it is really helpful to understand what is going on, and sometimes the only option is REST.
For the latter I thoroughly recommend POSTMAN if you need to interact, though Azure is also improving native API exploring support. I discovered POSTMAN through an azure Friday video with Steven Lindsay who has some really really useful modules on GitHub. This is really helpful for CosmosDB (Documentdb as it was) and really helped me debug some Gremlin issues.
Next is the PowerShell module for CosmosDB which sits over REST and as well as being an awesome example of the kind is also a really helpful module for checking interactions with CosmosDB.
And finally an excellent post by Michał Pawlikowski on connecting to Azure Data Lake Storage with REST API with PowerShell, showing awesome detective work and publishing a bunch of really useful cmdlets.
Kubernetes and AKS in particular is becoming more and more important to us at work. In our experimental facility we have to stand up varying compute platforms; my main project is examining a specific workload on HPC and part of it needs Kubernetes to support some supporting work.
Then I stumbled across a blog by Chris Johnson . I’ve met Chris (officially a “good guy”) exactly twice in Person; once in 2010 in Berlin at an Ignite Session (when Ignite was a smaller scale effort) for SharePoint 2010 where he presented a session on Microsoft Certified Master, and secondly at Ignite in Orlando last year when I made a point of catching him before he presented a session of the Microsoft Cloud Show with Andrew Connell (also officially a “good guy”) and Julia White (yes, that Julia White).
Anyway, this is one of those posts which is as much for my benefit as yours!
Working in the Microsoft cloud ecosystem (ok, Azure) and working for a Microsoft Partner steers me heavily towards the tools that the vendor provides. This works on a number of levels; mainly around depth of knowledge and personally this means getting ready for the next exam.
For code and script storage this means Azure DevOps and GitHub, the choice has got harder lately due to the tweak to the “free” tier on GitHub and private repos but we all love Azure DevOps because of pipelines and all the other stuff, even though my primary day to day use is as a Git Repo.
Of course I’ve been using Visual Studio for years and the online version for as long as it exists. The rebrand to Azure DevOps also brought a new url option going from <org>.visualstudio.com to dev.azure.com/<org> and the latter has created some new joy. I really recommend Multi-Factor authentication and love using the latest and greatest tech from Microsoft including their security features as it’s about the only way to keep up with the threats we face out there on the internet.
Of course it comes back to bite you from time to time and this morning has been a classic case. The current Git for Windows Release is 2.21.0 but a key component for me as a multi-factor protected user of Azure AD and Azure DevOps is the Git Credential Manager for Windows and there are a bunch of fixes relating to the new dev.azure.com url in version 1.19. Git for Windows 2.21.0 unfortunately includes Git Credential Manager for Windows 18.104.22.168 so you’ll need to install in strict order to get this the correct way round.
My symptoms included the following:
- No prompt for credentials when cloning my repo, just a couple of http errors then a prompt for a password.
- No prompt for credentials even though I had removed the pat tokens and emptied Windows Credential Manager.
- Errors thrown at the Git level (I tend to live in VS Code or Visual Studio).
One of the (many) great aspects of my current role working in the Innovations area of a UK Bank is a relentless introduction to new features in Microsoft Azure. At my stage with Azure in practice and exams it is usually a new feature or behaviour that has dropped as part of a generation 2 (E.g. Storage vs Data Lake) or evolution of features or more subtly a change to the defaults of a combination (e.g. Automation and Desired State Configuration Extension). Then there are the “never heard of it” moments when a term gets mentioned and I rattle straight to a search engine.
One of these a few months back was Azure Cyclecloud, one of our projects involved input from Microsoft and their HPC specialist proposed it as a key component of the platform being evaluated. In our case it is acting as an orchestrator / scheduler and keeping tabs on a handful of low priority virtual machine scale sets.
I’ve not had any direct exposure to HPC beyond awareness due to Microsoft architectural exams I’ve done in the past for on-premises Windows, and latterly Azure cloud. The good news is within parameters that the news is good and Azure CycleCloud appears straightforward and being predominantly IaaS based is fairly easy to secure within our patterns. My thoughts so far are:
- The web admin interface is fairly sensitive to environment – I’ve lost about a day to Internet Explorer (doesn’t work) and the reverse proxy on our firewall appliances mangling page scripts.
- The manual install is straightforward and reliable in my limited experience – we have a vnet model that it sits in quite nicely and the documentation is good on required ports and cluster communications.
- Azure Cyclecloud being HPC and batch etc comes from open source land, so lots of command line and Linux – quite ironic that my career includes so many loops (my first job at an accountants in the 1980s included being the guy who wrote sql reports using vi on a unix practice management system).
- Following on from the previous point, Azure Cyclecloud integrates with Active Directory and therefore has it’s own RBAC model – very important to understand if you are trying to secure it.
- I have a few concerns about the quickstart deploy, mainly due to the public ip address bound to a server but that probably reflects our use cases and my background. (Googling “cyclecloud initial setup” reinforces this concern as a number of servers in initial setup pop up).
- The cloud account relies on a fairly big service principal so it’s important to keep on top of that bearing in mind the last two points.
- About 50% of the time I get the name wrong and call it Azure CloudCycle. This hit rate is slowly improving.