I think, therefore I am. I am, therefore I sail

Tag: Devops

Photo by Josh Sorenson on Unsplash

How to Make a Domain Proxy for Digital Ocean Spaces

Digital Ocean has been my go to solution for hosting for many years. When the Spaces service, an S3 compatible object store, was introduceed, I jumped on board right away. The service performs well and allows me to manage all the web infrastructure from one location.

The drawback with Spaces, to me, is how custom domains are handled. It is possible to do, but you have to turn over DNS control of the domain to Digital Ocean. That is not always possible or practical to do. For a couple years I have run various sites with the domain provided by Digital Ocean.

A default Spaces domain has the format: account.datacenter.digitaloceanspaces.com
For my personal blog this looks like: lobaugh.sfo2.digitaloceanspaces.com

A useable, but not very attractive domain. I decided to revisit the topic.

Nginx has some powerful proxy capabilities in it, and it turns out that it works quite well to create a domain proxy.

With the proxy enabled, visiting
https://assets.lobaugh.net/image.png
Will return the file from
https://lobaugh.sfo2.digitaloceanspaces.com/image.png

Running my own domain proxy does introduce additional complexity and slight overhead, but I am comfortable with it.

I will present the nginx.conf file in its entirety here, then walk through it below.

log_format upstream '[$time_local] Requested: $host$uri - Proxied $proxy_host - Response time remote $upstream_response_time request $request_time';proxy_cache_path /tmp/nginxcache levels=1:2 keys_zone=my_cache:10m max_size=2g 
inactive=600m use_temp_path=off;server {listen 80;location / {
proxy_cache my_cache;
proxy_pass https://lobaugh.sfo2.cdn.digitaloceanspaces.com$uri$is_args$args;
add_header X-Proxy-Cache $upstream_cache_status;
access_log /dev/stdout upstream;
}
}

That’s it! The configuration is fairly simple.

* log_format upstream — (optional) Establishes the format of the log file. Not needed if logging is disabled. Turning off logging may help performance.
* proxy_cache_path — Configures the nginx caching of the files from Spaces. A 10 minute cache, with a max of 2 gigabytes is created. Though not necessary, this will help save on server resources and wait time for clients.
* server.listen — Establish the web service
* server.location — Configure the web service
* proxy_cache — Sets up the previously configured cache
* proxy_pass — This is the meat and potatoes. This passes the call to assets.lobaugh.net to the Spaces service, and retrieves the file
* add_header — (optional) Adds a simple header item that allows us to inspect whether the response was cached. Can be safely left out
* access_log — Send the output of the log to /dev/stdout, based on the upstream format. Not needed if logging is disabled. Turning off logging may help performance

This is v1 of the configuration. Nginx provides a lot of neat options that can tweak and optimize it. To learn more about the options, here are a few helpful links:

https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/ 
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
https://dev.to/danielkun/nginx-everything-about-proxypass-2ona 
https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching
https://www.nginx.com/blog/nginx-caching-guide/ 
https://dev.to/shameemreza/accelerate-your-website-with-nginx-as-a-reverse-proxy-cache-a9o

I am satisfied with this setup for now. It has allowed me to achieve the custom domain I wanted, and has minimal performance impact.

Picture of tables

How to extract a single table from a MySQL sql file

Ever received a large MySQL dump file and only needed one table out of it?

I ran across just this issue. The full database took significant time to import and I only needed a small subsection of the DB, just one single table. Rather than waiting for it all to import, I pulled together a sed command that extracts a single table from the sql file. Check it out:

sed -n ‘/Records of TABLE_NAME/,/Table structure for/p’ FULL_SQL_FILE.sql > table.sql

Replace TABLE_NAME and FULL_SQL_FILE with the names of the table you want and the sql file to pull it from, respectively.

You will now have a table.sql file in your directory that is ready to be imported.

Photo by chuttersnap on Unsplash

Powered by WordPress & Beards