Automating with Python and Boto3

I worked as a professional software engineer for many years. First I wrote C code and built it using make. Later I wrote Java code and built it originally using Ant and then Maven. I’d written lots of code which used the AWS Java SDK. At the other extreme, I’d written BASH shell scripts, including some which used the AWS CLI tools. I had never considered automating with Python and Boto3.

An organization was migrating to AWS and asked about automation and Infrastucture as Code. The Java approach was great for “Enterprise Applications”, but the build process introduced considerable turnaround time. The BASH approach provided instant feedback, but using the output from one command as the input to the next command was often awkward, and sometimes impossible.

Automating with Python and Boto3


A coworker suggested that I try automating with Python and Boto3. At first, I was horrified. A language which forced indentation to declare statement blocks? I thought that died with COBOL!

What I grew to love was the ability to write code without needing a build system. It was easy to call AWS APIs and use the results. I could find libraries to do almost anything that I needed. AWS Lambda supports Python.

I also realized that I always indent my Java code. The curly braces in Jave were pretty much redundant.

The Python Way

There were other things to get used to. I was informed that these were “The Python Way”.

The most significant of these was about exception handling. In Java, code was carefully constructed not to throw exceptions. In Python, exceptions were used as a form of flow control. The whole idea of the try … else block was frightening.

The Python way did make things more efficient. Instead of making a call to AWS to determine state, and then another call to perform an action, Python would using a single call. The try…else construct adds some elegance to the approach. It allows the statement that might cause the exception to be independent of the code to run if there was no exception.

An Example Python and Boto3 App

The Boto3 library allows Python to talk to AWS. The following program will use the credentials setup by the aws configure CLI tool to connect to the default AWS account and list all the Amazon Simple Storage Service (S3) buckets. Lines 4 & 6 do the real work:

 

Share
2 comments
  1. Except that the documentation for all this is almost non-existent. Sometimes an error message will be buried in the response dict and no exception raised, or there will be an exception, but there is little or no documentation on what these will look like, or what ones can occur; so one ends up simulating errors to figure it out. Its messy as hell.

    • Without specifics, it’s difficult to respond directly to your comments.

      Code often seems to be written to deal with “the happy path” first and exceptions are only handled when they are encountered. The example about just displays the error and continues, but it only exits anyway. If a recoverable error is ever experienced, a fix could be added to repair and retry. It’s very reactive, but meets the “minimal viable product” concept that is very much in vogue these days.

      Also, there is always a question of whether a particular condition is an exception or normal operation. For example, describing a named resource which doesn’t exist. Should an exception be raised, or a nil value returned (which could itself lead to an exception if not handled correctly).

      I’d like to hear more about specific issues. AWS is also usually quite accommodating when it comes to fixing issues or making improvements. Have you contacted them?

Add Comment

Required fields are marked *. Your email address will not be published.