SSH forwarding vs deployment keys

There comes a time in the life of every web application when it has to get deployed and be made accessible for the masses. Deployment, among other steps like setting up the database and configuring the web server, comprises primarily of getting the source code to the server. Many prefer creating an archive of the source code on the local system and copying it over to the remote machine. Though this is the simplest method, its horrors start to show up with the increase in the size of the code base and the number of developers in the team.

There comes a time, when keeping the people taking care of deployment need to be sure that the source code on the server is also the freshest one from the master branch of the source version control system. Services like github and bitbucket provide a utility called deployment keys - What one needs to do is generate a public-private key pair and add the public key to the repository as a 'deployment key' for the project. This makes it possible to clone / pull the source directly from the repository. But when there are more than one server that the source code needs to be deployed to - well either a public key from each of those servers is added as deployment key. - or the private key is copied to the ~/.ssh/ directory on all the servers.

There is a way to avoid all that work - by enabling SSH Forwarding. The concept is pretty simple. When a user logs in to the server remotely via SSH, the session on the remote server is able to access the private keys of the user's workstation for accessing other machines through this one. This is also the basic concept behing SSH tunnelling.

All one needs to do is modify ~/.ssh/config file and add something like:

Host anuvrat.in
    ForwardAgent yes

or

Host ip.add.re.ss
    ForwardAgent yes

or

Host *.yourcompany.com
    ForwardAgent yes

The benefit is that all the remote machines that you ssh into, provided they match the host parameter can now authorize themselves against github / bitbucket and get the source code.

references:

Thoughts: Selfish Happiness.

In the bollywood movie "Om Shanti Om", Shah Rukh Khan quite famously said:

"itni shiddat se maine tumhe paane ki koshish ki hai ki har zarre ne mujhe tumse milane ki saazish ki hai....
kehte hai agar kisi cheez ko dil se chaaho toh poori kaynath tumhein usse milane ki koshish mein lag jaati hai"

The quote is mostly misinterpreted, by ignoring the first half in urdu. Atoms in the universe don't conspire get you what you desire. Forget the atoms, you don't even surface in the thought processes of most of the humans in your circle of acquaintances. And those few, who actually do spend a few seconds thinking of you, don't go beyond speculating your opinion about them.

The very hard to accept truth is, among 6 billion humans and trillions of other species, YOU are all alone on this planet - which in turn is a lonely speck of dust circling one of the billions of stars in the universe. We as a species have been so severly scared of this loneliness, that every single human civilization has very successfully imagined the existence of an omnipotent, omniscient, all powerful supreme being to comfort their insignificant, insecure, timid selves. Basically, they were wisphering "All is well" to themselves. But that is matter for another post.

Most of the time we spend introspecting, we think about "my problems", "my wishlist", "my grudges", "my crushes", "my weekend", "my blah", "my bleh". The point I am trying to make is we are very self-centered and quite selfish - each and everyone of us. At this alligation, the years of moral indoctrination of "selfishness being bad" makes our innards scream:

"Well, I am not selfish, I think of my friends before myself" - Oh, but you don't. What you care about MORE, is "Your friendship" with them.

"I am not selfish, I help my friends" - Sure you do, but is that truly altruistic? Would you feel nothing, have no grudges against the friends you helped, if they turned their backs when you need them? Let that sink in for a while.

"Oh, but what expectation can I have an old lady I helped cross a road the other day?" - Well, some people help others to feel good about themselves - still counts as being selfish in my book.

"I give to charity - what can I possibly hope to get back from it?" - Charity? Even if you didn't advertize your deed on facebook, didn't brag about it to a single soul, didn't think highly of yourSELF for a millisecond and forgot about it instantly - may I request you to consider helping people learn to earn their living instead of letting them have it without effort. Wasn't there something about teaching a man to fish?

Contrary to what has been preached in sermons and taught in Moral Sciences, being selfish is not bad. Not nearly as bad as faking hypocritical selflessness while knowing deep down that actually we are. What is not good, and may at times even be dangerous, is being dilusional of one's true nature. One of my acuaintances, who relentlessly tries to help people around him and has nurtured an image of himself being selfless, has on multiple occasions ended up losing friends - friends whose growing expectations outgrew his selflessness.

Most of the universally accepted good things people do is because they are selfish. May be we can go a step further and not only accept but also expect people to be human and hence selfish. Doing so will enable us to lower our expectations from people around us. Hadn't Siddharth Gautam said something about not having expectations being the key to happiness, after becoming the Buddha?

Would your opinion about your colleagues be any different if you allowed them the luxury of being selfish and committing mistakes as they too are flesh and bone beneath their skins? Most of the people you are working with are in the job for the money and some for the kick.

The folks running the government and also human - if we entitle them to have some selfish desires we will realize that whatever they do, they do in order to ensure their reelection into the office - secure the prospects of their future selves. Why should they pay any heed to the overflowing drains and potholes on the street of your locality, unless you make it an inescapable hurdle in the process of their getting reelected? What was that about "not asking what the country has done for you"?

We complain about degrading standard of education, but people running schools and colleges are not doing so with the altruistic intention of educating every single homo-sapien on the planet - why should they work any harder unless students demand for better tuition and instead of flocking around for purchasing the degree, refuse to pay for substandard education. What was that about "being the change"?

In my opinion what SRK's dialog should be interpreted as is - "if you really want something, you have to want it so much, that you MAKE even the atoms conspire to get it for you". The difference is, it is you who goes to work for realizing your dreams.

Travelog: Unplanned weekend excursion

In February this year, I along with my colleagues, visited Rishikesh, while we were out on the pretext of attending another's marriage. The breathtaking beauty got me addicted, and I knew I had to get back soon.

The NGO classes which usually keep me busy on Sundays are having a session break and I didn't have anything critical to be taken care of on Saturday. I decided to get away from the city for the weekend. At 5 in the evening on friday, I booked a ticket for Haridwar via redbus. Got home by 9, packed a bag with essentials and a towel [never hitchhike without one ;)] and was at the bust stop by 11. The bus left from Kaushambi within half an hour and the next morning I found myself in Haridwar.

I tried freshening up in a Sulabh but the cleanliness got better of me. Went to a hotel nearby, paid a hundred bucks for using one of the rooms or bathrooms primarily. Once that was taken care off, I just as randomly called one of my college friends, who lives in Dehradun and had invited me long ago, when the weekends weren't so ... mine. I said I was in Haridwar and was wondering whether to go to Rishikesh or come over to Dehradun. In another 30 minutes I was no a bus going to Dehradun. I hadn't ever been to Doon before, let alone his house and he had gone to sleep after a long work night and was not picking up my calls. Stranded at I ISBT, fruitlessly trying to call common acquaintances I had almost made up my mind to go to Rishikesh when I thought of searching for the name of his tech-startup that he is currently running out of his place. And voila, google uncle told me his address.

The long night ride on what they call an AC Delux bus (which is in no way comfortable) had me exhausted and I slept through the afternoon. Later in the evening we went out for a short excursion on my friend's uncle's old bajaj scooter ... met another friend and went further for a short ride on Mussoorie road. On Sunday, we had planned to visit Raipur but had to cancel that and instead we caught a bus to Mussorie. Walked along the Mall road, having burger and paranthas at non descript small shops. Discussions about random tech stuff over coffee with a priceless view of the hills was just amazing ... reminded of the hackathons we had in hostel.

With promises (more to myself) to be back soon I left from there and just managed to catch my bus. And here I am, back in my apartment, typing what to the best of my knowledge is my first travelogue.

On gzipping files and serving them.

Even though configuring servers is an integral part of my life as a web dev, there are some nuances which elude me every now and then. Recently I tried to serve some gzipped files via nginx and hit a road block. The problem statement was simple, I had some files that I had compressed using gzip and saved on the disk of my server, coz my files took 80% less space on the disk compressed than when they were not. And because python makes gzipping so simple I am practically compressing every dump-file and saving on space on the server and time transferring it. Here is how,

import gzip
my_data = "<insert your favourite content>"
with gzip.open( "my_awesome_file.txt.gz", 'w') as f:
    f.write(my_data)

Similarly the file can be opened in 'r' mode if the aim is to read it. Sorry, I got carried away, coming back to the point of serving them via nginx. I had a problem, so I asked uncle google. However, every result that he came up with was regarding compressing and then serving static files and in my case the files were already compressed. After some tinkering I did figure out my mistake. I was setting up the content-type as 'gzip', which the browser was not able to display and hence was prompting to download as binary file. Where as what I had to do was specify content-encoding as gzip. Here is the snippet that helped me get it done:

location /cache/ {
        alias /path/to/the/directory/cache/;
        expires     max;
        add_header  Content-Type text/html;
        add_header  Content-Encoding gzip;
        add_header  Cache-Control public;
        add_header  Last-Modified "";
        add_header  ETag "";
}

I have a couple of small instances on Digital Ocean where I run my twisted based crawlers. And more often than not, there is a need to look at the page that was downloaded when it was downloaded, in order to determine whether there is an error in the parser or the page actually did not have the information that was supposed to be extracted. Since these instances had virtually nothing except Twisted on them, I was reluctant to install apache / nginx just to occasionally serve some cached html pages. Initially, I wasn't compressing them and they ended up consuming the entire disk space of 20 gigs. I found that twisted's built in web server was enough to serve the files:

twistd web --path /path/to/web/content

Soon the disk space proved to be very insufficient, flushing the files became a pain. Though a simple cron job or a fabric script would have rid me from the drudgery of running the rm command on each of those crawlers, I didn't quite feel like doing that. Looking for an alternative, I found gzip ... which allowed me to compress the htmls to almost 90% in some cases and 70-80 percent on an average using the script I illustrated above. Then there was a new problem, tiwsted web couldn't serve the gzipped files right out of the box. This meant I had a problem ... so ... yes you guessed it right, I went to uncle google. After shuffling through the results I was able to stitch up this small script that serves the gzipped html files just fine and with the proper content encoding they are also automatically uncompressed by the browser after being downloaded. Without much ado, I present to you the script:

# to be saved with an .rpy extension
# eg serve_something.rpy
from twisted.web.resource import Resource

class MyResource(Resource):
    is_leaf=True
    def render_GET(self, request):
        request.setHeader("content-type", "text/html")
        request.setHeader("content-encoding", "gzip")
        print request.args
        filename = request.args["id"][0]
        return open("/path/to/directory/contaning/cached/%s"%filename).read()

resource = MyResource()

This needs to be run using following command :

twistd --pidfile=tweb.pid web --path=/path/to/the/directory/having/the/rpy/file --port=8888

And it works like a charm, making me one happy python developer. :)

Troubles while moving /var to other partition

I am currently running ubuntu 13.10 on my office wala laptop. Since last couple of days the root partition was proving unsufficient. A simple df showed that /var was the partition consuming most of the space. Little more tinkering revealed that the databases I had on my laptop for development environments were the primary source of trouble. Adding to that were the various snapshots stored by docker. Obviously, the crudest solution, would have been to repartition while reinstalling the os. However, the task of configuring everything again was daunting, and laziness got the best of me. Yesterday, out of the blue, it dawned on me that I had ~30G of unpartitioned space for future me's desire of trying out newer distros. Thence, came the idea of mounting /var on another partition.

cfdisk and mkfs.ext4 created and formatted the new partition /dev/sda8, but beyond that I was clueless. So naturally, I asked google for the steps. And the results it came up with were:

  1. mounting the new partition elsewhere, (say /var2)

    mkdir /var2; mount -vt auto /dev/sda8 /var2
    
  2. copying everything from the present /var to /var2, using rsync -rvtlD for coping the files properly

    rsync -rvtlD /var/* /var2/
    
  3. editing /etc/fstab to mount the new /dev/sda8 as /var automatically upon everyboot, by adding the following line

    uuid="long-hexadecimal-string-printed-by-blkid" /var ext4    errors=remount-ro 0       2
    
  4. rebooting the computer

    sudo reboot
    

I followed the guide, blindly, however, it didn't quite work. Ubuntu failed to load the graphics settings. I could see that the new partition was indeed mounted on /var via

mount | grep sda8

executed on tty1 (ctrl+alt+f1). For a second there, I thought that maybe I would have to reinstall the os, after all. I restored the backup of /etc/fstab I had taken earlier and rebooted. My mental peace returned when the system booted properly. I had a problem and I was stuck, so I went to uncle google, again. However, no one had reported having lost their display settings because of mounting /var on other partition. I left it and went to bed. Today, I thought of checking the output of ls -l for /var and /var2. The premissions were identical however the user and groups were not. A bit of googling followed. And I came across some post that suggested using rsync -avrtlgoD. It worked.

References

[1] http://askubuntu.com/questions/39536/how-can-i-store-var-on-a-separate-partition/

[2] http://www.digizenstudio.com/blog/2007/06/14/watch-out-when-you-move-var-in-ubuntu/

[3] http://explainshell.com/explain?cmd=rsync+-rvtldgoD

[techlog] January First Week

Due to popular demand from my humble reader base I am listing the technologies I played around with.

Polished the way I deploy my web apps. Nginx + Supervisord + (uwsgi / gunicorn) + (django/flask) has been my stack for quite some time now. Lately I figured, that having supervisord and uwsgi installed by the operating system's package manager in its own way kind of scatters the app around. Log files can be configured to be where ever you want but almost all the awesome tutorials / blog posts tell you to have the configuration files in /etc itself. Though its not a very big deal to symlink them, why do it when you can have everything around your project directory itself? It also helps in lowering the chances of lack of permission related errors. And my favourite part is that I can put relative paths in all configuration files and be sure that it will work both on my laptop and production ... without much ado.

Came across the awesomeness of python-eve and flask-admin. Eve is a REST api framework for mongodb. I used it for one of my small projects at work to immensely reduce my load. My task was to handled the backend and provide required APIs. If it hadn't been for eve I would have found myself writing a different api for every little request that came my way. But with eve, almost every thing that the consumers of my api wanted could be achieved with little modification to their request parameters ... sorting, filtering, referencing ... name it and it was there.

Thats it for today, stay tuned for more. Adios.

Simplest useful flask script

Usually django is my first choice when the task at hand is a web project. I have also used django for some projects which weren't actually meant to be web applications and been benefited by the plethora of utilities that come along with it. However, every once in a while you come across simple one time tasks for which django feels like a huge unnecessary monster bloatware. Something similar happened today.

So there is this improperly documented third-party API that was to be integrated into one of the applications of our company. And this API provides a callback feature, such that when you send it an asynchronous request, it would after some time, post some data back to a URL you provide. Hovever, the format and structure of the data that would would be posted is not documented and the point of contact of the API provider is also not sure about it.

Other endpoints of the api which required just a GET or a POST request were easily explored and integrated with the help of Postman. (the awesome chrome plugin and python-requests). But this callback needed a url where it would post data and I didn't quite feel like tampering the code of any of the existing hosted applications. Hence came this small and elegant python script, which does nothing more that printing whatever was posted to it.

In django, its quite a cumbersome task, if all you want to do is this. So here I am, sharing a silly little piece of code that simplified my life today.

Requests and PyQuery

Off lately tasks at work involved downloading web pages and extracting some information out of them. Mechanize and BeautifulSoup worked wonderfully initially. Though after having come across Requests and PyQuery it seems that I have been doing it wrong (read tedious, non pythonic way) all along. So most of the code written in past few months will have a massive overhaul.

What seems excellent today will be superseded by something better tomorrow. Evolution is inevitable which makes learning an infinite loop with no exceptions.

This post was an attempt to log a bump in my learning cum evolving curve.

contacts replace specs

I had been prescribed spectacles at the age of 9 and had been wearing them ever since. The glasses went on getting thicker following the steady decrease in my ability to see without them. In the mean time, I was suggested contacts quite often, but just like a few other whims, of mine I had determined that I would do so with my own earnings, which I did, today.

At the optician's,  when I clearly saw my spectacle-less face in the mirror, through contact lenses it dawned on me that I was doing so after ages. And that I did not even remember what my face looked like without the specs.

August LUG meet

Whether you know or not, there exists, in Delhi/NCR, a totally non-secretive, and open to all community, consisting of students, free-lancers, entrepreneurs, engineers or just individuals who are geeks at their hearts. The Linux User Group of Delhi. Sure, different people call it by different names in different areas eg ILUGD, IITD-LUG, ALiAS, JMILUG but the crowd that comes forward upon hearing the call for a meeting from any of those is essentially the same, besides some new faces who are kindled and welcomed with open arms.

The geeks are naturally in front of their computer screens most of the time and are almost always connected via facebook groups, google groups, mailing lists, IRC channels etc, however, they have now made it a tradition to meet in person at least once a month. Such meets are appropriately called Geek Meetups and members from in and around Delhi gather at one location to interact, network, share knowledge by means of short informative talks, or hunt for talent / jobs etc.

Such a meet was organized and hosted within the premises of my Alma Mater, by the members of its Linux club called the Amity Linux Assistance Sapience (or ALiAS in short). So naturally, I was there.

I have been to college a couple of times after the concluding ceremony, you know the one after which you are no longer required to pay any attention to nagging issues like classes, assignments, examinations or even going to college for that matter. So yes, I had visited the campus a few times in the recent past and experienced the most extraordinary and contradictory of emotions: gloomy happiness, joy with a tinge of sadness ... A smile involuntarily appears on my face while a sinking feeling reminds that none of the faces that I was so accustomed to find looming in various corners are there to be seen. Familiar land strange inhabitants.

But then I met the members of ALiAS, and the sinking feeling seeped away. It was the first time that the present team of ALiAS had organized such an event - after we the seniors departed. I was surprised to find myself relaxed and unworried while they handled the event management issues -which used to be my headache an year ago. After witnessing the excellent manner in which they planned, tackled and solved every detail, hurdle and problem, I am sure that the club will not only thrive and survive but will become the legend to which we will refer with reverence.

To the present team of ALiAS: Well Done, Good Show, and all the very best for your future endeavors.