Rubén Varela

sftp - Basic commands

2015-12-14T10:19:05-05:00

Connect via sftp to host:

sftp user@host.example.com

After connecting, some of the basic and most used SFTP commands:

ls           # list the content of the remote directory
lls          # list the content of the local directory
pwd          # print the current directory in the remote host
lpwd         # print the current directory in the local host
cd           # change directory in the remote host
lcd          # change directory in the local host
mkdir        # create a directory on the remote host
lmkdir       # create a directory in the local host
get filename # Download *filename* into current local directory
put filename # Upload into the current

Quickly downloading a file:

sftp user@host.example.com:path/to/file

Symlinks utility

2015-12-14T08:29:17-05:00

symlink is great for identifying different characteristics of symbolic links. It will list if a symbolic link is broken (dangling), if it's absolute or relative and if it's pointing to another partition.

If you specify it, it will also remove broken/dangling symbolic links and/or change absolute links to relative ones.

First, we compile it:

OS X:

curl -O http://www.ibiblio.org/pub/Linux/utils/file/symlinks-1.4.tar.gz
tar xzf symlinks-1.4.tar.gz
cd symlinks-1.4/
export CFLAGS=-I/usr/include/malloc; make

If you want, you can install it by running make install. In this case, I need it to be part of a script, so I will leave it in this folder for now.

If you run the binary without specifying anything, this is the output:

 /V/M/t/symlink_test 
 😎  𝛌 ~/Downloads/symlinks-1.4/symlinks
symlinks: scan/change symbolic links - v1.3 - by Mark Lord

Usage:  symlinks [-cdorstv] dirlist

Flags:  -c == change absolute/messy links to relative
    -d == delete dangling links
    -o == warn about links across file systems
    -r == recurse into subdirs
    -s == shorten lengthy links (displayed in output only when -c not specified)
    -t == show what would be done by -c
    -v == verbose (show all symlinks)

Now, this is how it works:

I created 3 symbolic links:

abc is broken. It references something that doesn't exist.
ruben is linking to my $HOME directory in another partition.
tmp is linked to a folder in this partition.

This is how you can use the utility:

 /V/M/t/symlink_test 
 😎  𝛌 pwd -P
/Volumes/MiniDrive/tmp/symlink_test
 /V/M/t/symlink_test 
 😎  𝛌 ls -lhA
total 24
lrwxr-xr-x  1 ruben  staff     4B Dec 14 08:27 abc -> /abc
lrwxr-xr-x  1 ruben  staff    13B Dec 14 08:34 ruben -> /Users/ruben/
lrwxr-xr-x  1 ruben  staff    23B Dec 14 08:35 tmp -> /Volumes/MiniDrive/tmp/

And this is the output you get without any parameters:

 /V/M/t/symlink_test 
 😎  𝛌 ~/Downloads/symlinks-1.4/symlinks ./
dangling: /Volumes/MiniDrive/tmp/symlink_test/abc -> /abc
other_fs: /Volumes/MiniDrive/tmp/symlink_test/ruben -> /Users/ruben/
absolute: /Volumes/MiniDrive/tmp/symlink_test/tmp -> /Volumes/MiniDrive/tmp/
 /V/M/t/symlink_test 
 😎  𝛌

You can easily redirect this to a file and grep/sort based on what you need to find.

Symlinks source
Symlinks software map

Recovering tmux when it's socket was deleted

2015-12-09T17:14:20-05:00

When you use tmux, it stores a socket so you can regain access to the detached sessions in the tmux server. These can be located in multiple places depending on your environment.

First tmux checks if an absolute path is given via -S.
If not specified, it uses the flag -L to name the socket.
The location where this socket is created is defined by $TMPDIR if set.
If not, it will be created under /tmp

This is all documented under tmux's man page

Reminds me of mktemp.

Now, if $TMPDIR isn't set, it will create the socket in /tmp. Sometimes this folder is cleaned and you can loose access to your tmux server. If this happen, your process will still be running, but you simply won't be able to connect.

If you run tmux list-session you will receive an error message:

$ tmux list-session
failed to connect to server: Connection refused

When you check ps aux | grep -i tmux or top or htop you can definitely see that tmux is running. If you run lsof -U you can also see the socket that tmux has open.

Now, reading through tmux's man page, you can find the answer. Under -L it says:

If the socket is accidentally removed, the SIGUSR1 signal may be sent to the tmux server process to recreate it.

Now, I also checked OpenBSD's man page and it adds a bit more of information:

If the socket is accidentally removed, the SIGUSR1 signal may be sent to the tmux server process to recreate it (note that this will fail if any parent directories are missing).

If it doesn't work, check the parent directories.

So, solution. If you're only running one tmux server, you can do:

killall -SIGUSR1 tmux

If you want to target a specific instance of tmux, you can use kill and specify the process ID.

If you want the numeric value, you can use kill -l to get a list. In the machines I have access (OS X and Linux), this is 10, but I've checked references online for other OS' and it can change.

Installing latest siege from source

2015-11-18T17:36:30-05:00

Siege is a utility for stress testing web applications and web servers. It's used to help expose vulnerabilities or failures that happen under stress.

The instructions for installing siege from source aren't too dificult. Here I'll be installing it on my home folder in an folder named opt.

In Debian/Ubuntu or other systems like it, you should install the package build-essential

 sudo apt-get update && sudo apt-get install build-essential

On OS X, you need the XCode Command Line Tools.

xcode-select --install

Now to install it.

If you want the latest version, you can just download it,

curl -O http://download.joedog.org/siege/siege-latest.tar.gz

Now we extract it,

tar xf siege-latest.tar.gz

It will create a folder named siege-x.y.z. In my case it's siege-3.1.3. We now cd into this directory:

cd siege-3.1.3

Run configure with the --prefix we want.

./configure --prefix $HOME/opt

Now we simply run make and make install

make
make install

If you want to do it via Git:

Clone the repo:

git clone https://github.com/JoeDog/siege.git

Change into the new siege folder
```
cd siege
```
Run the bootstrap script.
```
utils/bootstrap
```
Now we can run configure with a prefix:
```
./configure --prefix ${HOME}/opt
```
Now we can make siege
```
make
```
And now we install it:
```
make install
```

Installing latest git without sudo privileges

2015-11-18T14:19:35-05:00

If you already have an previous version of Git installed, you can clone the latest version:

git clone https://github.com/git/git.git
cd git

Now, we can actually build git

make

By default Git will be installed onto ~/bin. Now we install it.

make install

If the folder doesn't exist, it will be created. The only thing we need to do now is make sure that ~/bin exists in your $PATH

echo $PATH
  /home/ruben/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin

If it doesn't, you can add it.

nano ~/.bashrc

Replace nano with whichever editor you want. At the end add:

export PATH=/home/<username>/bin:${PATH}

To exit nano

ctrl+o
ctrl+x

Note

If you want to change where Git is installed to, for example, maybe /home/ruben/opt, we can substitute the make step with:

make prefix=/home/ruben/opt

Obviously replacing ruben with your username. This should work for everyone:

make prefix=${HOME}/opt

For extra options, there's an INSTALL inside the Git folder.

Install Julia - Nightly version on Linux

2015-11-18T12:29:16-05:00

Clone the repo
git clone https://github.com/JuliaLang/julia.git
Make sure your version of Git supports --detach. If not, you'll run into this issue
```
cd julia
git checkout --detach
```
If it does support --detach, keep going. If not, upgrade Git. Check this guide on how to do it without privileges
Now we run make.
```
make
```
We can use -j X where X is a number (number of cores for example) to specify how many commands make can run simultaneously. If we specify -j without a number, make will not limit the amount of processes that can run simultaneously.

Everytime I've tried to build Julia, I've gotten an error towards the end. The last time the error was:

\033[33;1m*** Clean the OpenBLAS build with 'make -C deps clean-openblas'. Rebuild with 'make OPENBLAS_USE_THREAD=0 if OpenBLAS had trouble linking libpthread.so, and with 'make OPENBLAS_TARGET_ARCH=NEHALEM' if there were errors building SandyBridge support. Both these options can also be used simultaneously. ***\033[0m
Makefile:1131: recipe for target 'build/openblas/libopenblas64_.so' failed
make[1]: *** [build/openblas/libopenblas64_.so] Error 1
Makefile:78: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

The good thing is Julia is very well built. Doing what it says, has always worked for me. In this case:

make -C deps clean-openblas
make OPENBLAS_USE_THREAD=0 OPENBLAS_TARGET_ARCH=NEHALEM

Remember to add -j X if you want.

Once this finishes, there will be a binary named julia in this folder.
Now you can create a symbolic link in any folder that is specified in your $PATH
```
echo $PATH
  /home/ruben/bin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin
```
In this case, I'll create the symbolic link under /home/ruben/bin which is defined already. mkdir $HOME/bin ln -s $(pwd)/julia ${HOME}/bin Since I don't know where you put the julia folder we cloned using Git. $(pwd) will give us that path (the path of the current folder to be more specific). And since I don't know the username you use, ${HOME} will give us that.

Optional: If for some reason /home/<username>/bin is not defined and you don't have access to write to any of the other folders, you can always add that to the $PATH variable.

    nano ~/.bashrc

Replace nano with whichever editor you want.
At the end add:

    export PATH=/home/<username>/bin:${PATH}

To exit nano

    ctrl+o
    ctrl+x

Access service on remote server (forward port to local machine)

2015-11-17T11:05:53-05:00

TL;DR:


Remote Port we want forwarded	11138
Local Port where we want it to be accessible	11139
Server hosting the service at 11138	remote.example.com

ssh -L 11139:localhost:11138 remote.example.com

The good thing about this is that the connection is secured/encrypted by SSH. Sometimes we can have a development server or a service that doesn't support encryption and we need to access it securely. Having the service only listen to local connections doesn't expose it to the internet and with something like the code above, we can access it securely from another computer.

Background: Talking with some graduate friends at Purdue a few months ago this came up. They needed help accessing their research on a remote server securely. This was a quick solution I gave them.

Check for broken links on a site

2015-08-03T00:00:00-04:00

Crawl rubenvarela.com¶

Get all results, even if okay (--verbose)
By default it will only return values that are errors or warnings.
You can omit the --verbose argument if you want.
Store the results in XML format, linkchecker-out.xml

In [141]:

!linkchecker -F xml http://rubenvarela.com --check-extern --verbose

30 threads active,     0 links queued,    4 links in   1 URL checked, runtime 1 seconds
30 threads active,    10 links queued,    7 links in   4 URLs checked, runtime 6 seconds
30 threads active,   124 links queued,  306 links in  39 URLs checked, runtime 11 seconds
30 threads active,    97 links queued,  333 links in  47 URLs checked, runtime 16 seconds
30 threads active,    59 links queued,  371 links in  60 URLs checked, runtime 21 seconds
30 threads active,    28 links queued,  402 links in  67 URLs checked, runtime 26 seconds
28 threads active,     0 links queued,  432 links in  75 URLs checked, runtime 31 seconds
13 threads active,     0 links queued,  447 links in  81 URLs checked, runtime 36 seconds
 0 threads active,     0 links queued,  460 links in  87 URLs checked, runtime 41 seconds

In [142]:

import xmltodict
from random import sample as randSample

In [143]:

with open('linkchecker-out.xml') as f: 
  obj = xmltodict.parse(f.read())

Explore the object.
- Get the main element and see it's content.
- Get 5 random elements and print them in order to see the objects.

In [144]:

for x in obj['linkchecker']:
    print x

# print type(obj['linkchecker'])
# print obj['linkchecker']['@created']
# print type(obj['linkchecker']['urldata'])

randset = randSample(obj['linkchecker']['urldata'], 5)
randset

# print obj['linkchecker']['urldata'][0]
# print obj['linkchecker']['urldata'][1]
# print obj['linkchecker']['urldata'][2]
# print obj['linkchecker']['urldata'][3]
# print obj['linkchecker']['urldata'][99]
# print obj['linkchecker']['urldata'][100]
# print obj['linkchecker']['urldata'][101]
# print ''
# print obj['linkchecker']['urldata'][100]['parent']['#text']

@created
urldata

Out[144]:

[OrderedDict([(u'url', u'http://getbootstrap.com'), (u'name', u'Bootstrap'), (u'parent', OrderedDict([(u'@column', u'13'), (u'@line', u'162'), ('#text', u'http://rubenvarela.com/')])), (u'realurl', u'http://getbootstrap.com'), (u'extern', u'1'), (u'dlsize', u'11937'), (u'checktime', u'7.068993'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://getbootstrap.com'), (u'name', u'Bootstrap'), (u'parent', OrderedDict([(u'@column', u'13'), (u'@line', u'261'), ('#text', u'http://rubenvarela.com/ana-g-mendez-university-system-agmus.html')])), (u'realurl', u'http://getbootstrap.com'), (u'extern', u'1'), (u'dlsize', u'11937'), (u'checktime', u'7.068993'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html'), (u'name', u'AAA Subastas (PRASA Data Bids)'), (u'parent', OrderedDict([(u'@column', u'21'), (u'@line', u'83'), ('#text', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html')])), (u'realurl', u'http://rubenvarela.com/aaa-subastas-prasa-data-bids.html'), (u'extern', u'0'), (u'dltime', u'0.000377'), (u'dlsize', u'9439'), (u'checktime', u'5.711945'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'http://rubenvarela.com/theme/css/pygments/native.css'), (u'parent', OrderedDict([(u'@column', u'5'), (u'@line', u'31'), ('#text', u'http://rubenvarela.com/museo-de-arte-de-puerto-rico-puerto-rico-art-museum.html')])), (u'realurl', u'http://rubenvarela.com/theme/css/pygments/native.css'), (u'extern', u'0'), (u'dlsize', u'4732'), (u'checktime', u'2.197381'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))]),
 OrderedDict([(u'url', u'/images/SUAGM/old/01_-_SUAGM_-_2015-05-25_21.55.28.png'), (u'name', u"'Old Home Page'"), (u'parent', OrderedDict([(u'@column', u'4'), (u'@line', u'147'), ('#text', u'http://rubenvarela.com/ana-g-mendez-university-system-agmus.html')])), (u'realurl', u'http://rubenvarela.com/images/SUAGM/old/01_-_SUAGM_-_2015-05-25_21.55.28.png'), (u'extern', u'0'), (u'dlsize', u'1165284'), (u'checktime', u'10.259280'), (u'level', u'2'), (u'valid', OrderedDict([(u'@result', u'200 OK'), ('#text', u'1')]))])]

Now to extract links with errors:¶

For each URL,
If the result is not okay,
Check if the valid and parent keys exists.
If they do, then print the values.
If the necessary keys don't exist, print the object.

In [145]:

for x in obj['linkchecker']['urldata']:
#     print x['valid']['@result']
    if not x['valid']['@result'] == "200 OK" :
        if 'valid' and 'parent' in x: #if keys exists in x
            print "Result Code: " + x['valid']['@result']
            print "Link location: " + x['parent']['#text']
            print "Line: " + x['parent']['@line']
        else:
            print "Couldn't find all elements:"
            print x
        print "Links to: " + x['url']

In [ ]:

Ana G. Mendez University System (AGMUS)

2015-07-14T00:00:00-04:00

Developer

Develop the website for the Ana G. Mendez University System. One of the main challenges they faced was with accessibility. They also wanted to incorporate the ability for all content to be translated between Spanish and English.

Technologies: Drupal, SSH, Git, Bootstrap, MySQL, Apache, PHP, jQuery, JavaScript, etc.

Systems

The first development environment for this website consisted of a VPS that had three environments, Integration, Staging and Production. In order to isolate each environment as much as possible, 3 instances of Apache where deployed.

Each environment was it's own git repo. This allowed for easy deployment by simply pushing to the correct remote.

Eventually SUAGM wanted to move to Acquia and everything was deployed there. The default structure Acquia uses was pretty much the same so deploying the code here was as simple as adding another remote and pushing the code.

Website implemented

New Website

New Home Page

New News Listing Page

New Single News View

New University Listing

New University Detail

New Section Home

New President Detail Page

New Vice Presidency of Marketing and Student Affairs

New Interactive Network Student Services

New Administration Page

New Career Planner

New Events Page

Old website

Old Home Page

Old News Listing Page

Old Single News View

Old University Listing

Old University Detail

Old Section Home

Old President Detail Page

Old Vice Presidency of Marketing and Student Affairs

Old Interactive Network Student Services

Old Admissions

Old University Important Links

Old Events Page

Museo de Arte de Puerto Rico (Puerto Rico Art Museum)

2015-07-13T00:00:00-04:00

Developer

Develop the new online system for the Puerto Rico Art Museum

This platforms needed to be fully bilingual and allow them to manage the different areas of the museum:

Permanent Collection
Exhibitions
Artists
Including their biographies and Artwork
Artwork
Including its Collection and related artwork
Calendar
Information pages

Homepage

Visita menu

Arte menu

Museo menu

Admision, Hours and Parking

Events Calendar

Join

Guided Visits

Collection

Artwork in the collection

Artist in the collection

Artist in the collection, expanded section

Puerto Rico Aqueduct and Sewer Authority Data Bid (AAA Subastas)

2015-07-12T00:00:00-04:00

Developer

Design and implement the PRASA Data Bids platform for the Puerto Rico Aqueduct and Sewer Authority.

Technologies: Drupal, PHP, MySQL, Apache, jQuery, (X)HTML, CSS.

Systems

Create the stack, strategies and guidelines to maintain the servers and backups.

Technologies: Ubuntu Linux Server, Apache, PHP, MySQL, rsync, ssh, git, Bash scripts, Python

Homepage

Project Dashboard

Project Documents

PG Engineering Solutions, PSC (PGES)

2015-07-11T00:00:00-04:00

Home Page

Slideshow transition

Transition after

Firm Overview

Message from the President

Services

Services - Hover

Project List

Project - Department of Housing

Project - Department of Housing - Gallery

Project - Department of Housing - 1st Image

Project - Guaynabo City Projects

Client List

PGES Online Systems

News

Contact Us

Cambiar background Mac OS X desde el terminal

2013-01-28T00:00:00-05:00

Estuve este fin de semana pasado buscando como adaptar un script que utilizo en Linux para Mac OS X (10.8 Mountain Lion para ser mas específico).

El script lo que hace es descargar una imagen y colocarla como background. Las imágenes las descarga de Reddit. Específicamente de /r/earthporn donde hay fotos impresionantes de lugares.

En mi caso, el script funciona en Python, el problema que tenia era que no sabia como cambiar la imagen de fondo desde un script. La solución es sencilla (una vez la encuentras),

defaults write com.apple.desktop Background '{default = {ImageFilePath = "/Users/.scripts/background_changer/image.png"; };}'

Un script sencillo, en Bash, que cambie el background seria

#!/bin/bash
###########################
#Author: Ruben Varela Rosa
#Created: 2013-01-28
############################

PathToImages="/Users/ruben/.scripts/background_changer"
Image="image_name.png"

if [ -f $PathToImages/$Image ]; then
    defaults write com.apple.desktop Background '{defaults = {ImageFilePath = "$PathToImages/$Image"; };}';
    killall Dock;
fi

Si desean cambiar el Background para otro usuario, tendrían que hacerlo como root. No mucho mas complicado, pero si hay unas cosas adicionales que hay que hacer.

#!/bin/bash
user="user_name"
path_to_images="/User/$user/.scripts/background_changer/"
images="image_name.png"

/usr/bin/defaults write /Users/$user/Library/Preferences/com.apple.desktop Background '{default = {ImageFilePath = "$path_to_images/$images"; };}'

chown $user /Users/$user/Library/Preferences/com.apple.desktop.plist

Espero que otros también encuentren útil lo anterior.