Posts

Intro You have all your data ready to go. It has been cleansed and transformed to all your needs. Now you need to make it easy to consume by your target audience. In this scenario our users can be just about anyone. We need to make the data easy to access and even easier to navigate. Challenge accepted! Tooling As the title suggests we are going to be using Streamlit to present our data. If you aren’t familiar with Streamlit I suggest you take a tour of their website or GitHub repo to see what they have to offer. I chose Streamlit because it offers the ability to get a small project like this off the ground very quickly. We will also be using Pandas, and this will all be running on Python 3.12.2. ...

Intro True story, the town I live in recently had some issues around the town budget. As I started to get involved I wanted to look at the data provided by our Board of Finance. To my misfortune, I found myself trying to analyze data in a PDF. Trying to do any sort of analysis on it was time consuming and frustrating. To make the data useful I needed to extract data from PDF and into something more flexible, in this case a simple CSV will do. Here is a sample of what we are working with: ...

Intro In our last post we used SQLAlchemy to bulk load data into SQL Server. Now that we can bulk load data we want to perform other actions on our database. In todays post we want follow a common loading strategy and TRUNCATE the target table before loading it. Adding this functionality to our script leaves us with a choice to make in how we want to execute our SQL statements and manage database transactions. ...

Intro You have a bunch of data and need to load it into SQL Server using Python. With the abundance of options out there its difficult to know what the best tool is to use. Lets approach the problem with two requirements. The solution needs to be easy to implement and blazing fast. Tooling Initially bcpandas looked like a great solution for this task. While it does fit our two requirements, it becomes very difficult to work with for data that contains multiple separators or any odd strings. If the data you work with is like mine it is almost never perfect. I quickly pivoted away from this due to its inability to handle imperfect data. ...

Intro Back in August HashiCorp switched Terraforms license from Mozilla Public License v2.0 (MPL 2.0) to a Business Source License. In response, the open source community has forked the last MPL release of Terraform to a new project named OpenTofu. To understand their mission checkout the The OpenTofu Manifesto. Current State OpenTofu has an alpha release out if you want to take it for a test drive. I did so myself with a few sample projects and they all worked without any issues. Biggest difference so far is using tofu apply instead of terraform apply in the CLI. ...

Intro With the exciting new announcement for Terraform Tests I couldn’t help but take them for a test drive. For our use case we will build on a prior post Terraform Azure: Reusable SQL Database Configurations. In the last post we created a module to deploy SQL Servers. Today we will add a new default prefix to SQL Server name and then build a test to check the name of the server. ...

Intro In this post we will cover how to import an existing Azure SQL Database to Terraform. If the resource wasn’t originally created by Terraform you will need to import it before Terraform can manage it. This comes in handy when you have existing infrastructure and want to start using Terraform later on. For a video walkthrough check out my YouTube video. Here we will use the standard import functionality provided by Terraform. There are other 3rd party tools you can use for this process as well. One of the most popular is Terraformer, I will cover how to use that in a later post. ...

Intro In this post we are going to build on the previous post Terraform Azure: SQL Database. We will cover how you can use Terraform to create multiple Azure SQL Databases with Terraform modules. We are going to turn the original Terraform into a module so we can create as many databases as we need with identical configurations. Another advantage here is maintenance, if a setting needs to be adjusted you can simply adjust the module and it will propagate to each resource using the module configuration. ...

Intro In this post we are going to deploy a SQL Server VM in Azure. Deploying a VM requires a little more work than the SQL Database we deployed in the last post. We will need to add some networking resources and configure the host VM. If you need help setting up Terraform please see my prior post, Terraform Azure: Setting Up Your Environment. This post assumes you have an Azure account. If you do not, you can create one for free. ...

Intro In this post we are going to cover how you can use Terraform to create an Azure SQL Database. If you need help setting up Terraform please see my prior post, Terraform Azure: Setting Up Your Environment.This post assumes you have an Azure account. If you do not, you can create one for free. We will take a slow approach to accomplishing our task. The goal here is to get familiar with Terraform and its commands by creating some basic resources. Terraform also has some great resources for getting started here. ...