Find Locking/Blocking Redshift Queries

https://aws.amazon.com/premiumsupport/knowledge-center/prevent-locks-blocking-queries-redshift/

#!/bin/bash

#################################################################################
# findlockblocks.sh
#
# Dead-stupid script that leverages existing RS queries and does a mashup that reports
# the current running queries that are blocking others, sorted by time running.
#
# Nice, simple way to see if there's actually a problem or if RS is just swamped.
#
#                
# TODO: Utilize a sourced bash file with usernames and passwords, as with other scripts                   
# in the library.                    
#                    
#################################################################################

RSUSER="vacasaroot"
RSPASS="xxxxxxxxxx"

read -r -d '' "stats_sql" << 'EOF'

SELECT
  a.txn_owner,
  a.xid,
  a.pid,
  a.txn_start,
  a.lock_mode,
  a.relation AS table_id,
  nvl (TRIM(c."name"), d.relname) AS tablename,
  a.granted,
  b.pid AS blocking_pid,
  DATEDIFF(s, a.txn_start, getdate ()) / 86400 || ' days ' || DATEDIFF(s, a.txn_start, getdate ()) % 86400 / 3600 || ' hrs ' || DATEDIFF(s, a.txn_start, getdate ()) % 3600 / 60 || ' mins ' || DATEDIFF(s, a.txn_start, getdate ()) % 60 || ' secs' AS txn_duration
FROM
  svv_transactions a
  LEFT JOIN
    (SELECT
      pid,
      relation,
      granted
    FROM
      pg_locks
    GROUP BY 1,
      2,
      3) b
    ON a.relation = b.relation
    AND a.granted = 'f'
    AND b.granted = 't'
  LEFT JOIN
    (SELECT
      *
    FROM
      stv_tbl_perm
    WHERE slice = 0) c
    ON a.relation = c.id
  LEFT JOIN pg_class d
    ON a.relation = d.oid
WHERE a.relation IS NOT NULL;

EOF

echo "$stats_sql" > /tmp/stats.sql


BLOCKPIDS=`cat /tmp/stats.sql | PGPASSWORD=${RSPASS} psql -h  warehouse.vacasa.services -p 5439 -U${RSUSER} -dwarehouse | cut -d'|' -f9 | grep "\S" | egrep -v "blocking|rows|\-\-\-"| sed 's/ //g' | sort -u`

#construct IN clause

for PID in $BLOCKPIDS
do
PIDCLAUSE="$PIDCLAUSE,$PID"
done
PIDCLAUSE=" AND pid in (${PIDCLAUSE:1}) "


stats_sql="select pid, duration/1000000 as seconds, trim(user_name) as user,substring(query,1,200) as querytxt from stv_recents where status = 'Running' and seconds >=1 ${PIDCLAUSE} order by seconds desc;"

echo $stats_sql > /tmp/stats.sql


echo "

Current active Redshift queries that are blocking the execution of other queries (may or may not be critical)
=============================================================================================================
"


cat /tmp/stats.sql | PGPASSWORD=${RSPASS} psql -h  warehouse.vacasa.services -p 5439 -U${RSUSER} -dwarehouse 

Sum Total Number Of Rows Affected By Queries

Stupid command line tricks: I often run SQL jobs that can take many days to fully execute, and I always pipe the error and stdout to a file. As such, you end up with lots of lines like this in the output:Query OK, 81218 rows affected (52.76 sec)Here’s a silly one-liner that will quickly total up ALLLLLL the rows that have been affected. (this assumes the output is being directed to a file named do_archive.out)

grep affected do_archive.out | cut -d' ' -f3 | perl -nle '$sum += $_ } END { print $sum'

Want to format your number with commas?  Pipe it through this:

sed ':a;s/\B[0-9]\{3\}\>/,&/;ta'

Installing Homebrew On A Mac

Advanced Mac users may appreciate using the Homebrew package manager, which greatly simplifies the process of installing command line software and tools on a Mac.

For example, if you want to easily install favorite command line tools on a Mac like cask, htop, wget, nmap, tree, irssi, links, colordiff, or virtually any other familiar unix command line utility, you can do so with a simple command. Homebrew downloads and builds the package for you.

This is obviously aimed at more technically savvy Mac users who spend a lot of time at the command line. While there’s no particular issue for novice users installing Homebrew on their Mac, the odds of novices finding it useful are slim, unless they intend to embark on learning the command line environment. Contrast that to power users who practically live in a terminal environment, whether longtime Mac users or migrating to the platform from the Windows or Linux world, who will immediately see the value of Homebrew.

Requirements for Installing Homebrew on Mac OS

prerequisites to installing Homebrew on a Mac include the following:

  • A Mac running Mac OS X 10.10 or later, though earlier versions are sort of supported
  • Command Line Tools must be installed on the Mac (either independently or through Xcode)
  • Knowledge of the command line and using bash or zsh

Assuming you’re interested in installing Homebrew and meet those requirements, then the rest is equally straight forward.

How to Install Homebrew on Mac OS

The simplest way to install Homebrew is through ruby and curl, accomplished with a single command. This approach is the same for installing Homebrew in all supported versions of Mac OS and Mac OS X.

  1. Open the “Terminal” application, found in /Applications/Utilities/
  2. Enter the following command into a single line of the terminal:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

  1. Hit Return and you’ll see a series of lines about what the script will install and where, hit Return again to agree or hit Escape to cancel
  2. Enter the administrator password (required by sudo execution) to begin installation

Installation of Homebrew will take a while depending on the speed of your Mac and internet connection, as each necessary package is downloaded and installed by the script.

When complete, you will see an “Installation successful!” message.

Now you’re ready to install software packages through Homebrew, or you can read the help documentation with the following command:

brew help

Installing Software Packages through Homebrew on Mac

Installing packages with Homebrew is super easy, just use the following syntax:

brew install [package name]

For example, to install wget through Homebrew you could use the following syntax:

brew install wget

Simple, easy. Once complete you can run wget as usual.

Run Queries On All DB Instances

Assumes there is a text file, “instances.txt” that has the IP or symbolic name of the server where the instances are Just put the query in quotes at the end of the script (such as pasting the below into runonall.sh)

params=$@
           
for i in `cat /home/rbyrd2/bin/instances.txt` ; do echo "$i:  " ; mysql -h $i -N -B -e "$params" ; done