How I managed my way through Presigned Url (GCS)

As a Mobile Developer, You get assigned to a task where you have to upload files to a Cloud Storage at first it sounds Easy Peasy

because at first thought it looked like a simple Multipart upload of file to a Backend server and the rest will be handled by the server but no, things were different and much more than only a simple multipart upload.

Presigned Url

A presigned URL is generated by a Cloud Storage (GCS or S3) for the user who has limited time access to the object, the expiration time of the presigned url is defined in the generation process of the url. The generated URL is then given to the unauthorized user. The presigned URL can be entered in a browser or used by a program.

A presigned url consists of following things:

• Base Url i.e (https://storage.googleapis.com/)
• Bucket Name i.e (test_bucket)
• File Name i.e (cat.png)
• GoogleAccessId (test-container@21test-value-123456.iam.gserviceaccount.com)
• Expires (1611664897 in milliseconds)
• Signature (Signature is a tricky part. Most of the working of presigned url works on a valid generated Signature string)

Let’s talk about the Signature string, The signature string is a composition of multiple items.

• Http Method i.e (PUT,GET…)
• Content MD5
• Content Type
• Expiry Time
• Bucket Name
• File Name

A plain Signature string looks similar to this:

PUT
UjvlJB6JqkYcYn01GTyRkQ==
png
1611664897
/test_bucketcat.png

But this is not the string we need to send as a value of Signature parameter. This plain string is converted and generated as Base64 encoded string, the final generated presigned url looks like this:

https://storage.googleapis.com/test-bucket/poui728sjfdejnobwapxz6gwhtj0?GoogleAccessId=test-container@99empirical-state-123456.iam.gserviceaccount.com&Expires=1611664897&Signature=12d21d21d12d12d12d12d12d12d12d12%2F7vmWbDcOcGytyA0V0wPMnwt6NSC2YJCKDYqep4VraB869rQ6Jy4e0UxTn7YY1dK4jefw%2BvycC9VHhnMJmFjWlXV1Ak9%2BJwMP4zjubjfbiMWd%2F9kPefosdfjiosdiofjsidjfio9290dasco90Hiwtfh4dCx7GTf949FWlccn7N%2B8PU6gkmMQ9btstjCh8YJLBDWQYvaW8%2F%2FRdvwwxzPsxuKqW6PjCIC9vbRmQO5cbDle%2BgLMFhByUGKRJehwgbgjfF%2F1ZM1GB6%2BpLK7pelB98f%2BJusWuDhaejqPU7mAALPStrY4zrOUgSsdw%9D%9D

The actual problem Ihad while working with presigned url was this error :(

<Error><Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your  Google secret key and signing method.</Message>
<StringToSign>
PUT 1611664897 /test_bucket/cat.png</StringToSign></Error> 

This error became a nightmare for me and I had to a lot of time and many useless attempts until I found a proper way to get rid of this error. It is a very common error that most of the developers out there face while working with presigned url, as the error statement is pretty straight forward that there is an issue with the generated signature string and it is not matching with the signature calculated by GCS.

The solution to this error is not consistent, It can always be so dynamic and tricky which makes it so hard to understand that where the things are going wrong. As I already mentioned that signature string consists of multiple items. That string is then converted to a Base 64 encoded string. Now the string which we’ve generated at our end should be similar to the string generated at Google Cloud Storage’s end. And if it’s not the same on both ends, your request is never considered a valid request at Google Cloud Storage’s end. Okay too much talking now. Let’s jump straight to the solution.

Long story short I was not generating the presigned url on my end. I was getting it from the backend server. There is a library for Rails called Active Storage which was doing the generation of presigned url for clients. The presigned url was fine but the actual issue was the Upload of file from mobile side. The PUT request, yeah the culprit was me not the Backend :(

The issue was in the Http Put request for uploading the file to storage. I was sending the content type header which was causing the error.

val signedUrl = signedUrlResponse.directUpload.url
val requestBody = file.asRequestBody()
val request: Request = Request.Builder().url(signedUrl).put(requestBody)
    .addHeader(HEADER_CONTENT_TYPE, file.getMimeType())-Wrong header
    .addHeader(HEADER_CONTENT_LENGTH, file.length().toString())
    .addHeader(HEADER_CONTENT_MD5, file.getBase64Hash())
    .build()

All I had to do was to remove this Content Type header from my put request to get it to working.

Next time you find yourself stuck with the API calls, make sure you’re only sending those headers which are required by the backend.

Happy Google Clouding.