Bug Troubleshooting w/ New Tools

Tech: Ruby, TigerConnect, Splunk, SSH, CSSHX, grep, curl, HTTParty

Challenge: an application is configured to log an event in Splunk after a successful send of a TigerConnect HIPAA-compliant alert message. The entry point is working, however the alerts are not being sent AND the event is not being logged. After successful troubleshooting to determine what’s NOT wrong, I turned to CSSHX and grep to scour the logs. Why CSSHX… our production instances run on 4 servers concurrently. I want to quickly navigate in 1 terminal tab w/ 4 windows and grep the the logs!

Code:

CLI (for 4 instances) => csshx username@dserver.location.extension username@dserver.location.extension username@dserver.location.extension username@dserver.location.extension

 

grep for various strings => grep -A 10 "error" log_file.log

-A # is for number of lines after the string

-B # is for number of lines before

 

I also wanted to check my HTTParty gem code that transmits the event from app to Splunk. I used a curl statement to mimic the HTTParty call.

CLI => curl https://url.extension:####/services/collector -k -H 'content-type: application/json' -H 'authorization: XXXXXX' -d '{"event":{"app": "data"}, "sourcetype": "_json"}'

-k (–insecure) allows for insecure server connections

-H header key, values

 

I was quite proud to implement the use of these tools when troubleshooting. I have used them in different contexts and it was cool to bring everything together to discover the issue. However (and sadly), the issue was much simpler.

Ruby ENV variable are strings. Even if the string is a “boolean”. 

in .env … VAR=false

if ENV['VAR'] == false then <do something> end

=> equates to FALSE… because ENV['VAR'] exists as a string, it is TRUE.