data-intensive

Book: Designing Data-Intensive Applications

December’s Book: Designing Data-Intensive Applications by Martin Kleppmann

Why this book: I want a deep dive into databases, and to think more along the lines of a software/systems engineer. 

Takeaway: this a huge, detailed textbook. I used a highlighter liberally. For this post, I want to create a quick list of common, critical questions & topics (basically, an outline!).

  1. Reliability, Scalability, Maintainability
  2. What is the load on the system (think bottleneck)?
  3. Relational Model vs. Document Model (RDMS vs. NoSQL)
  4.  What to index? 
  5. How to log?
  6. Compatibility – Backward, Forward
  7. Encoding… JSON, XML, Protocol Buffers, Avro, etc.
  8. Distributed Data – Scalability, Fault Tolerance/Availability, Latency
  9. Replication: single-leader, multi-leader, leaderless
  10. Replication – synchronous vs. async
  11. Partitioning (Sharding) combined with Replication
  12. ACID: Atomicity, Consistency, Isolation, Durability
  13. Dirty Reads, Dirty Writes
  14. Serialization 
  15. Distributed Systems problems: unreliable networks, faults/partial failures, timeouts, unreliable clocks, process pauses
  16. Consistency through Linearization
  17. Systems of Record vs. Derived Data Systems
  18. Batch Processing vs. Stream Processing

Pattern: Iteration within Interation (pt. 2)

Tech: Ruby

Challenge: returning to iterating! Last time, I wanted the sub-iteration to skip records already looked at, as well as skip the index forward. Let’s gooo!

Code:

i = 0

patients.each_with_index do |row, index|

next if i > index # will skip through rows already processed in the while loop

# using while equal makes it a little easier to understand than part 1’s solution

while patients[i][:id] == patients[index][:id]

# i will increase, and as long as it keeps matching our entry index :id; we are within the same patient’s records

# do logic

i += 1

break if !patients[i] # no more records

end

end

 

Takeaway: I previously implemented part 1’s solution, and this solution in production code. I wonder if there is a scenario where part 1 would be optimal comparatively (I think no because rows are redundantly processed).

vandelay

{React} importing & exporting

Tech: React

Challenge: while working through a tutorial, an error appears and no content is displayed!

react_import_export_error

I narrowed down the problem to the export/import of the tutorial’s js code. The tutorial is very basic, and my repo uses create-react-app, so several times I needed to troubleshoot errors that the tutorial didn’t run into since it was a different environment.
 
Code / Solution:

Here is the code supplied by the tutorial for importing the js code:

import {ReversibleList} from './ReversibleList.js'

I also saw my own code for importing React and Component:

import React, { Component } from 'react';

Why the {} sometimes, and other times not? Could this be the issue?
 
With a little help from the below article, I discovered the difference between export and export default!

If there are multiple exports, then the {} are needed. If the file uses export default, then no {}!

In code terms…

if (export default) {

// ex – export default class This extends React.component

import This from './This.js'

} else {

// ex – export class This extends React.component

import {This} from './This.js'

}

 

Reference:

https://medium.com/@thejasonfile/a-simple-intro-to-javascript-imports-and-exports-389dd53c3fac

https://github.com/lukekedz/react-armory-tutorial

 

Dockerfile: Command Placement?! (pt. 1)

Tech: Dockerfile, Sass, CSS, node, angular 1.5

Challenge: seeing curious error when building ng container

Error!

blog_dockerfile_1
 
Curious… let’s take a look at the Dockerfile…

blog_dockerfile_2

 
After not too much troubleshooting… the problem is the placement of

RUN npm rebuild node-sass

 
!! By simply placing the rebuild after line 12 ADD ., all is well. Part 2 coming… why?

HealthShare Exchange (HSX): API Connection

Tech: ruby, rails, dotenv, httparty, APIs!

Challenge: connect & query HSX’s API. Querying APIs is relatively new to me, and I wanted to capture the solution to a particular issue I encountered. The authorization POST required the params as a json body (specifically noted in the docs), but the GET required an URL query string (not specified in docs!). Basic, but was confusing for a minute.

Code:

auth with JSON body params

hsx_json_auth

provider query

hsx_get_query

 

Reference:

https://www.healthshareexchange.org/

 

turtles

Pattern: Iteration within Interation (pt. 1)

Tech: Ruby

Challenge: need to iterate over a set of patient records…within the set, each patient has several records… so when the initial iteration gets to a new patient, the code needs to include a sub-iteration specific to the patient. For example, a patient’s subset of data includes their location & treatment history. We want to capture the patient’s last location, but their first instance of treatment. The data is extracted into a patient-specific data hash.

Code:

patients.each_with_index do |row, index|

i = index
until patients[i]['ID'] != patients[index]['ID']

update_patient_hash(row['ID'], data)

i += 1
break if !patients[i] # no more records

end

end

 

Takeaway: this could benefit from refactoring…

  1. The sub-iteration repeats the processing of records due to reference to the row for data capture, instead of patient[i][data]
  2. Could the initial iteration’s index be reset/moved to the new patient once the sub-iteration is finished?

Ruby Template => Data to JSON

Tech: Ruby, JSON, Docker

Challenge: data for daily reporting is captured in a class instance variable hash. If/when the docker container restarts, I want the data, which is regularly written to a JSON file, to be reloaded into the @hash.

Code:

def initialize

if FileTest.exists?('./data.json')

@instance_var_hash = JSON.parse(File.read('./data.json'))

else

@instance_var_hash = { ... }

end

end

Potential Pitfall: when sandboxing, I had an instance variable hash with symbol keys. When reading in the JSON file and writing to my instance variable, I generated an error in my methods… :key in original instance variable versus 'key' from incoming JSON.
 
Does the JSON file exist?

if FileTest.exists?('./file_name.json')

Read file

JSON.parse(File.read('./file_name.json'))

Open the file & write

File.open('./temp.json', 'w') do |f|

f.write(JSON.pretty_generate(object))

end

 
Reference:

https://github.com/lukekedz/ruby_json_template

http://ruby-doc.org/stdlib-1.9.3/libdoc/json/rdoc/JSON.html

http://ruby-doc.org/core-2.2.0/FileTest.html

https://stackoverflow.com/questions/5507512/how-to-write-to-a-json-file-in-the-correct-format/5507535

https://hackhands.com/ruby-read-json-file-hash/

 

%@#$! wants to use the “login” keychain

Tech: mac, keychain

Challenge: not really a challenge… more of a reminder for something irritating every 3 months. Work mac requires a new password every 3 months, and the keychain does not auto-update.

Code:

go to finderApplicationsKeychain Access

on left… Keychains… select login

go to… menu barEdit

click… Change Password for Keychain "login"

enter old & new pw

 

Reference:

https://support.apple.com/en-us/HT201609