Solve Gcloud Terraform authentication issues

In this blog, I want to show an issue we encountered at a customer while working with Google Cloud and Terraform. We discovered that the Google SDK is having two different active logins. That is not completely clear at first sight.

User is not authorized

To start with the issue we had. We are working with a team of people working on a terraform project to be able to provision our Google Cloud environment. A colleague created a nice terraform configuration that I wanted to test and play with.

Before starting to add things to the configuration I wanted to check if the configuration file still reflects the Google Cloud infrastructure so I ran:

 $ terraform plan

This resulted in the following error:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

google_project.proj: Refreshing state... (ID: project-x-1)
Error refreshing state: 1 error(s) occurred:

* google_project.proj: 1 error(s) occurred:

* google_project.proj: google_project.proj: Error reading Project "project-x-1": googleapi: Error 403: User is not authorized., forbidden

Damn, let's see if I could reproduce the error with the gcloud SDK commands.

 $ gcloud projects list
PROJECT_ID            NAME                  PROJECT_NUMBER
project-x-1           Project x             123456789012
 $ gcloud projects describe project-x-1
createTime: '2017-08-04T12:43:01.088Z'
lifecycleState: ACTIVE
name: Project x
  id: '12345677889'
  type: organization
projectId: project-x-1
projectNumber: '123456789012'

That's weird: I'm able to see the project with the gcloud SDK, but not with terraform.

After a while a colleague found this in the Google Cloud console:


That's strange, I don't want to use my GoDataDriven account for the customer stuff.

Let's check my authentication:

 $ gcloud auth list

        Credentialed Accounts
*       <customer_google_account_here>

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

That seems to be correct. But let's check logout and login again.

 $ gcloud auth revoke
Revoked credentials:
 - <customer_google_account_here>

 $ gcloud auth login
Your browser has been opened to visit:

WARNING: `gcloud auth login` no longer writes application default credentials.
If you need to use ADC, see:
  gcloud auth application-default --help

You are now logged in as [<customer_google_account_here>].
Your current project is [None].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID

Wait a second, what is that WARNING??

Now the monkey comes out of the sleeve

There appear to be two different authentication sessions, the normal gcloud auth and gcloud auth application-default.

The gcloud commandline used the first, terraform uses the second.

After revoking the application-default session and login again with the correct account the sky was blue again.

 $ gcloud auth application-default revoke

You are about to revoke the credentials stored in: [/Users/rvanweverwi

Do you want to continue (Y/n)?  y

Credentials revoked.
 $ gcloud auth application-default login
Your browser has been opened to visit:

Credentials saved to file: [/Users/rvanweverwijk/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests
Application Default Credentials.

GoDataDriven is hiring

Stay up to date on the latest insights and best-practices by registering for the GoDataDriven newsletter.