Docker Cheat Sheet

No Comments

Isn’t documentation really helpul!? Working on multiple DevOps tools, I tend to mistype the commands or confuse one with the other. So here is my Docker cheat sheet –

  1. docker –help – Displays help menu
  2. docker –version  –  Displays docker version
  3. docker version –  Displays docker version for both Docker client and daemon
  4. docker image ls -a – Displays a list of all Docker images
  5. docker container ls -a – Display a list of all Docker containers
  6. docker rm $(docker ps -a -q)  – Removes all Docker containers in quiet mode
  7. docker rmi -f $(docker images -q) – Removed all Docker images in quiet mode

  8. docker run -it helloworld/1.0 sh – Propmts a shell to view the contents of Docker image

  9. docker build -t helloworld:1.0 .  – Build a docker image with the name “helloworld” and tag ‘1.0’. Pay attention to the ‘dot’ at the end of the command. It implies ‘Dockerfile’ in your local directory be used

  10. docker run -p 8888:5000 helloworld:1.0 – Instantiate the running instance of a Docker image ‘helloworld”. Port 5000 is exposed on the container and is mapped to 8888 on the host machine

  11. docker login – Authenticates a user to the cloud Docker public/private repository
  12. docker tag helloworld:1.0  username/myrepo:1.0 – Tag a local image to specific docker repository and the user

  13. docker push crkatz/myrepo:1.0 – Pushes the tagged image to the docker public or private repository

  14. docker container logs [OPTIONS] CONTAINERID – Tails the logs of the Docker container

 

 

Categories: Containerization, DevOps, Uncategorized Tags: Tags:

Set up Python Virtual Envs

No Comments

Developing an application in its own virtualenv is vital. Virtual envs help isolate python envrionments and enable you to install dependant packages in this isolated environment rather than globally.

  1. First find out the version of python you installed. I have python 2.7 default install on MacOS and python 3 via Homebrew package manager for MacOS
  2. Python packages required – virtualenv, virtualenvwrapper.
    pip3 install virtualenv virtualenvwrapper  #pip3 is using python3 interpretor
  3. Virtualenv package enables you to create project/application specific virtual env. Virtualenvwrapper is an extension of Virtualenv. It helps create,manage and delete virtual envs making it easier to work on more than one project at a time eliminating conflicts.
  4. Add the below lines to the ~/.bash_profile


    #tells which interpretor to use. In this case ‘python3’

    VIRTUALENVWRAPPER_PYTHON=’/usr/local/bin/python3′
    #set the env variable and assign a path to create projects

    PROJECT_HOME=’/Users/chetanak/Box Sync/Projects’

    #reload script
    source /usr/local/bin/virtualenvwrapper.sh

    #Exports the varible and makes available the previous and the new env to all the processes. All the virtual envs are stored under ‘.virtualenvs’ folder in your home directory
    export WORKON_HOME=$HOME/.virtualenvs

  5. Save the above file and run these commands

    mkdir ~/.virtualenvs

    #Load your ~/.bash_profile with the new changes.
    source ~/.bash_profile

  6. Now that all the paths and directories are created and set, you can create your virtual environments. Simply use –

    mkproject <projectname>

    #This creates a virtual env with the name of the project under ~/.virtualenvs directory and also create the project directory with the same name under $PROJECT_HOME. Since we are using the virtualenvwrapper functions, upon creation, it will activate the virtual env and ‘cd’ into the project folder

  7. To deactivate the virtualenv type ‘deactivate’ at the prompt
  8. If you want to know more about the virtualenvwrapper commands go to – http://docs.python-guide.org/en/latest/dev/virtualenvs/

Read More

Setting the ENV to run WLST scripts remotely

No Comments

1. Install Java, Python, Jython. Set path for each installation.

2. Download the Zip distribution for Oracle WebLogic Server only that is intended for Weblogic Server development. At the time of this writing, I have Weblogic server version 10.3.5. The zipped version is about 180MB and very light. Unzip the contents.

4. Inorder to setup the ENV you need to run setWLSEnv.cmd or .exe file depending on your OS. This file is located under your weblogic installation folder – “C:\wlserver\server\bin”

5. If you get errors while running file, then open the file setWLSEnv.cmd and make sure the paths that this file needs are set in your environment variables. If you don’t have them, set them and re-run the file. You should see a message confirming that your environment has been set.

6. To run your WLST scripts you need to set the env in the shell first. Followed by the command “java weblogic.WLST filename.py

Categories: Jython, Python, WLST

R Packages

 

  1. Hmiscrcorr() function in the package produces correlations/covariances and significance levels for pearson and spearman correlations. Input must be a matrix and pairwise deletion is used.
  2. pastecs
  3. psychThe package facilitates the summing the statistics by grouping variable
  4. doBy
  5. corrgramContains correlation visualization function
  6. latticeProvides options to condition the scatterplot matrix on a factor
  7. gclusProvides options to rearrange the variables so that the higher corralations are closer to the principal diagonal
  8. scatterplot3d
  9. rgl Creates interactive and spinning 3d scatterplots

Others R descriptive statistics functions include sapply(), fivenum(), summary()

corrgram(), pairs(), splom() are correlation visualizing functions

Categories: Packages, R-Stats

R-Color Palettes

No Comments

R has many built-in color palettes. They can be used as graphical parameters as below.

  1. col = rainbow(n)
  2. col = heat.colors(n)
  3. col = topo.colors(n)
  4. col – terrain.colors(n)
  5. col = cm.colors(n)

n is the size of the values being plotted. For more info enter the following help commands:

1. help(package=colorspace)

2. ?rainbow

Categories: Graphics, R-Stats

Data Science Books

No Comments
Categories: Big Data Sources

Data Sets

No Comments
Categories: Big Data Sources

My reference notes on Python

No Comments

1. Data types

  • int
  • float
  • string – can contain letters, numbers and symbols. String needs to be within quotes.
  • unicode – a special type of string
  • boolean – takes the values “True” or “False”
  • list – enclosed in square braces
  • dictionary –they have key value pairs. Pairs are enclosed in curly braces

2. Comments

# Prefix a hash for a single line comment

“”” Enclose multi   line  comments                                                                                                                                                       between triple quotation marks”””

3.  Arithematic Operations – +, -, *, /, **, %

4.  Comparision operators: ==, <=, >=, >, <, <>, !=

      Comparision operators results in a “True” or “False” values and the result of comparision can be stored in a variable.

5.  There are three boolean operators and the are listed here in the ORDER they are operated: not, and, or

6.  Broken String can be fixed by using the escape backslash!

  • ‘This isn\’t flying, this is falling with style!’

7.  String is like a List. A character in the string can be extracted by the index value beginning with 0 and starting from the left of the string. For ex.

varString = “Python”

# Print the 4th letter.

fourth_letter = “Python”[3]

print fourth_letter

8.  Slice a String or List, by using the index values

varString = “Python”

# Print the first 3 characters.

fourth_letter = “Python”[3]

print fourth_letter

9.  Built-in string functions

  • len()
  • str()
  • lower()
  • upper()
  • isalpha()

str() and len() can work on other data types. str() converts the data types to String.    lower()and upper() are used with a dot notation and can only work on strings

10.  built-in functions for int and float types

  • max()
  • min()
  • abs()

11. type(), is a buil-in function that outputs the type of the data type.

print type(13)

#The above prints <type ‘int’> to the console

12. Few examples for PRINT statements

# Print the concatenation of “Good Morning World”
print “Good” + “Morning” + “World”

string_1 = “Hello”
string_2 = “World”

print “Let’s be nice and say, %s %s.” % (string_1, string_2)

Use the % operator to replace the %s placeholders with the variables in parentheses. We need the same number of %s terms in a string as the number of variables in parentheses

13.  To accept the input from the console

name = raw_input(“Enter your name:”)

14.  A list of few Python modules

  • datetime
  • math
  • random

15.  Few syntaxes and declarations

  • define a function – def function():
  • Generic Import – import module
  • Function Import – from module import function
  • Universal Import – from module import *

16. Listing all the functions from a module after the import

import math

list_all = dir(math) # Sets list_all to a list of things from math
print list_all       # Prints all available functions

17.  List manipulation functions. Prefix each with the listname.

  • append()
  • extend()
  • index(item)
  • insert(index, item)
  • sort()
  • remove(item) – will remove the actual item if it finds it
  • pop(item) – will remove the item at index from the list and return it
  • del(listname[index]) – same as pop() but will not return it
  • join()
  • split()

18.  Dictionary items are deleted by – del keyword. Ex. del dictionaryname[key]

19.  range(), function is a shortcut for generating a list.

  • range(stop) –> range(5)
  • range(start, stop) –>range(0, 10)
  • range(start, stop, step) –> range(0, 4, 2)
  • my_list = [I, for I in range(11) if i%2 == 0] # This prints out the even numbers between 0 and 11
  • squares = [i**2 for i in range(1, 11)]  # Another way to create a list

20.  For loop syntax

  • for eachitem in listname:
  • for key in dictionaryname:
  • for i in range(len(list)):

21. List Slicing

  • my_list[start: end: stride]

Note that the default index starts at 0. The default end index is the last element in the list. Stride is the space between items in the sliced list. A positive stride value traverses the string from left to right. The negative stride value traverses the string from right to left.

22.  Python also allows functions to be passed as variables and values

print filter(lambda x: x % 3 == 0, my_list) 

# Here filter uses Lambda to determine what to filter. Lambdas are a quick way to create functions. But if you want to reuse a function, then it is better if you declare it by using “def

23.    Printing items in a dictionary

  • print my_dict.keys()
  • print my_dict.items()
  • print my_dict.values()
Categories: Python

Connecting R to Oracle Database

No Comments

The following three packages are required to connect to Oracle Database:

  1. DBI
  2. rJava
  3. RJDBC
    The  next steps:
    1. Need appropriate Oracle JDBC Drivers compatible with the Java version you like to support.
    2. To connect to the source database you need the following: Hostname or IP, Port, Service name or SID, Username, Password
    3. Use the information from Number 2 to construct a Data Source Name or DSN in the format:

    jdbc:oracle:thin:@hostname:port:sid

    Now, execute the following steps to connect to the database:
                                       

library(RJDBC)
# Create connection driver
jdbcDriver = JDBC(driverClass=&quot;oracle.jdbc.OracleDriver&quot;,
classPath=&quot;lib/classes12.jar&quot;)

#Open connection
jdbcConn = dbConnect(jdbcDriver,&quot;jdbc:oracle:thin:@database.hostname.com:port:sid&quot;,
&quot;username&quot;,&quot;password&quot;)

#Close connection
dbDisconnect(jdbcConn)
Categories: Data Sources Tags: Tags: , ,

R Data Types

No Comments

List

An immediate thing to remember is, a List is also called a Vector. A list is a collection of numbers, strings. A list is also a collections of lists. See below:

** A collection of numbers **

> a = c(1,2,3,4,5)   
> a   
[1] 1 2 3 4 5

** A collection of strings **

>b = c("Hello", "World!") # Character Vector. 
> b
[1] "Hello" "World!"

**A Collection of logical values **

>l = c(TRUE, TRUE, FALSE, FALSE, TRUE)  #Logical Vector
> l
[1] TRUE TRUE FALSE FALSE TRUE

** A collection of lists, strings, and numbers **

>f = c(a, b, 'Bingo') 
> f 
[1] "1"      "2"      "3"      "4"      "5"      "Hello"  "World!" "Bingo" 

> d = list(a, b, f)
> d
[[1]]
[1] 1 2 3 4 5

[[2]]
[1] "Hello" "World!"

[[3]]
[1] "1"      "2"      "3"      "4"      "5"      "Hello"  "World!" "Bingo" 

> e = list(First=1, Last='name', age=17.5)
> e
$First
[1] 1

$Last
[1] "name"

$age
[1] 17.5

** Identify the elements of a List using the convention [] or [[]] **

> f[1]
[1] "1"
> d[[1]]
[1] 1 2 3 4 5
> d
[[1]]
[1] 1 2 3 4 5

[[2]]
[1] "Hello"  "World!"

[[3]]
[1] "1"      "2"      "3"      "4"      "5"      "Hello"  "World!" "Bingo" > 

> e$age
[1] 17.5

Note: For strings, both the double and the single quotes work.

 

Factor

Factor is an important object type in R. In order to understand its usage and importance, you must first understand the difference between Categorical and Ordinal variable.

A Categorical variable, also known as Nominal variable, is classified into categories and there is no intrinsic ordering of the categories. For ex. Gender variable has two category values,  Male, Female, and but there is no agreed way of ordering the variables. Hair Color variable has different category values but cannot be ordered. Hence a categorical variable only allows you to assign a category but does not order the values in the ascending or descending order.

An Ordinal variable is similar to the Categorical variable, but the difference is Ordinal variable can be arranged in an order. For ex. Volume variable has the categories Low,Medium,High and can be arranged in an order.

In R terminology, the different categories are called Factors or Levels. A nominal factor is used to represent a nominal variable and an ordered factor is used to represent a ordinal variable.

R stores the nominal values as a vector of integers from [1..n], n being the total number of unique values in nominal variable, and an internal vector of character strings, which are the original values, is mapped to these integers.

The number of times a factor or a level appears for a given variable determines their frequency.

** hairColor as a nominal variable **

>hairColor = c(rep("Black", 5), rep("Brown", 10))
> hairColor
[1] "Black" "Black" "Black" "Black" "Black" "Brown" "Brown" "Brown" "Brown"
[10] "Brown" "Brown" "Brown" "Brown" "Brown" "Brown"
> hairColor = factor(hairColor)
> hairColor
[1] Black Black Black Black Black Brown Brown Brown Brown Brown Brown Brown
[13] Brown Brown Brown
Levels: Black Brown

> summary(hairColor)
Black Brown 
    5    10 
> attributes(hairColor)
$levels
[1] "Black" "Brown"

$class
[1] "factor"
#Internally R alphabetically assigned the values 1 = Black and 2 = Brown
#R now treats “hairColor” as a nominal variable

** volume as a ordinal variable **

>volume = c("small", "medium", "large")
> volume
[1] "small"  "medium" "large" 
> volume = ordered(volume)
> volume
[1] small  medium large 
Levels: large < medium < small
> attributes(volume)
$levels
[1] "large"  "medium" "small" 

$class
[1] "ordered" "factor" 
# R reorders the variable and associates 1 = large, 2 = medium, 3 = small 
# R now treats "volume" as a ordinal variable 

 

Data Frame

A Data Frame is like a matrix with rows and columns and is a collection of vectors. While each column has the same mode (numeric, character, factor, logical, etc.) and same length, different columns have different modes of data.

>a = c(1,2,3,4)
> b = c("Jack", "Jill", "Peter", "Mickey")
> c = c(TRUE, TRUE, FALSE, TRUE)
> dframe = data.frame(a,b,c)
> dframe
 a      b     c
1 1   Jack  TRUE
2 2   Jill  TRUE
3 3  Peter FALSE
4 4 Mickey  TRUE

> attributes(dframe)
$names
[1] "a" "b" "c"

$row.names
[1] 1 2 3 4

$class
[1] "data.frame"

> names(dframe) = c("ID", "FName", "Bool")
> names(dframe)
[1] "ID"    "FName" "Bool" 
> summary(dframe)
            ID          FName      Bool        
 Min.   :1.00   Jack  :1   Mode :logical  
 1st Qu.:1.75   Jill  :1   FALSE:1        
 Median :2.50   Mickey:1   TRUE :3        
 Mean   :2.50   Peter :1   NA's :0        
 3rd Qu.:3.25                             
 Max.   :4.00    

> dframe$ID
[1] 1 2 3 4
> dframe$FName
[1] Jack   Jill   Peter  Mickey
Levels: Jack Jill Mickey Peter
#FName is a Nominal variable
> dframe$Bool
[1]  TRUE  TRUE FALSE  TRUE

 

 

Tables

One way to create a table is using the table command.

One Way Table – A table with one row

From the above variable, hairColor

>hairColor = table(hairColor)
hairColor
Black Brown 
    5    10  <p>1&amp;#160; 1</p> <p>
>$dim
[1] 2

$dimnames
$dimnames$hairColor
[1] "Black" "Brown"


$class
[1] "table"
>summary(hairColor)
Number of cases in table: 15 
Number of factors: 1 

 

Another way to create a one way table is to create a matrix of numbers. The format to create a matrix is

 

mymatrix <- matrix(vector, nrow=r, ncol=c, byrow=FALSE, dimnames=list(char_vector_rownames, char_vector_colnames))

> mymatrix = matrix(c(2,3,4), ncol=3, byrow=TRUE) 
> colnames(mymatrix) = c("C1","C2","C3") 
> mymatrix
    C1 C2 C3
[1,]  2  3  4

 

Two way Table – A table with more than one row Note: All arguments must have same length

 

> color = c("red", "red", "blue", "pink") 
> shade = c("light", "dark", "dark", "light")
> mytable = table(color, shade)
> mytable
      shade
color  dark light
  blue    1     0
  pink    0     1
  red     1     1

Creating a table using a matrix of numbers

> myvector = c(1:20) 
> rownames = c("r1", "r2", "r3", "r4")
> colnames = c("c1", "c2", "c3", "c4", "c5") 
> mymatrix = matrix(myvector, ncol=5, byrow=TRUE, dimnames=list(rownames, colnames)) 
> mymatrix
  c1 c2 c3 c4 c5
r1  1  2  3  4  5
r2  6  7  8  9 10
r3 11 12 13 14 15
r4 16 17 18 19 20

 

Array

Arrays are similar to matrices but can have more than two dimensions. At the prompt, type help(array) for more information.

Categories: Data Types