Thursday, October 24, 2019

Multiprocessing to change class variables in python - experiments

I am trying to build a library to help compute certain numerical computations in a parallel fashion for certain data analytics tasks.

Python's standard multiprocessing module
Summary: works well, computation time comes down drastically compared to serial computation. Uses pickle to serialise the parameters being passed to the function.
Drawback is that the class variables remain at same init state or in the same state before multiprocessing begins.

Pathos' multiprocessing module
Summary: Faster than standard multiprocessing module; uses dill instead of pickle to serialise objects.
Drawback is that the class variables remain at same init state or in the same state before multiprocessing begins.

About to try Ray library (GitHub)

Tuesday, November 06, 2018

NGT installation for Mac OSX


This article captures the changes required to install the Neighbourhood Graph and Tree for Indexing on Mac OSX.
The GitHub repo is available here: https://github.com/yahoojapan/NGT

Initial NGT Installation


      $ unzip NGT-x.x.x.zip
      $ cd NGT-x.x.x
      $ mkdir build
      $ cd build
      $ cmake .. ## (for non-MAC)

For Mac OSX, the above command needs to be replaced with the below one. Also, need to install GNU cc/c++ from brew.

      $ cmake -DCMAKE_C_COMPILER=/usr/local/bin/gcc-8 -DCMAKE_CXX_COMPILER=/usr/local/bin/g++-8 ..
      $ make
      $ make install

      $ ldconfig

The above 'ldconfig' command works for Linux/Unix. For Mac OSX, it has to be:

      $ sudo update_dyld_shared_cache

NGT Python installation


$ cd NGT_ROOT/python
$ python setup.py sdist
$ pip install dist/ngt-1.2.0.tar.gz

For the above command, use the following command (after GNU cc/c++ installed)

$ env CC=/usr/local/bin/gcc-8 CXX=/usr/local/bin/g++-8 pip install dist/ngt-1.2.0.tar.gz


Wednesday, May 15, 2013

Opportunity cost behind every purchase

I learned (courtesy: an introduction to irrational behavior, coursera) a new perspective of  opportunity cost that can (in fact, it should) be applied to every single purchase we do. The concept is: 'if I make this purchase for 'x' money, what is it I will giving up in future?'.
Frankly, it took me a while to figure it out. But now that it has been dawning slowly upon me, I realized how important it can be in our decision making. The side effects are aplenty:
1. Makes us think a bit more on future spending.
2. Gives us a perspective how important is this purchase to the present time-frame
3. Compare our emotion or pleasure that we derive by making present purchase to something that can give much more in future,
4. Lastly, curbs impulsive purchases.
Many a time, step into a mall with no clear intention to buy or to-purchase list. Yet, step out of the mall with half dozen purchases, is pretty common phenomena.
On a experimental basis, I started creating and maintaining my to-purchase list. The list goes on the refrigerator listing out items (grocery and non-grocery) which are finished and need to be purchased.
When I started this exercise, I believed it should not make much of a difference, though I do indulge in impulsive spending occasionally and the amount splurged is not that big to take note. I believed I spend carefully.
Surprisingly, following this practice:
* firstly reduced the number of trips I used to make to purchase these 'necessary items'.
* secondly, I am amazed at how less is really required to purchase in comparison to what was being purchased earlier.
* most importantly makes me feel good about myself as I now purchase/consume responsibly. Avoids me to involve in over consumption and food wastage.
* lastly, how wrong I was in analyzing my spending patterns. I understand myself and my temptations better. But this list and opportunity cost twist curbs these temporary temptations and shows only real requirements.
Sometimes, we should be taking such courses. Why I chose to do this course is a long story. But to put it in short, I know I am always irrational when it comes to a lot of things. The title of the course was apt and I wanted to understand human irrational behavior.

Wednesday, March 06, 2013

Who says office space is a must!

http://www.businessinsider.com/successful-people-who-work-from-home-2013-3


Wednesday, August 22, 2012

Moving away from lists

Any and every online search displays results in lists. Most often, the lists are long, way too long. So that sets me thinking: we should have a new design artifact that replaces these lists. The artifact or plug-in should avoid endless scrolling and page navigations ('next page').

Google sometime back tried 'Wonderwheel'. I was excited about such a visual artifact, but have not seen it being successfully used. Probably it still lacked in its utility.

Sravanthi Kollukuduru, Kamalakar Karlapalem worked on concept of wheels to avoid scrollable options displayed when clicked on a select box. Aptly titled 'Wheels of Web', this is the beginning to move away from boring long lists.

And towards this challenge: I mine.

Monday, October 06, 2008

Versatile Artists' Filters for Variety in Music Playlists

I loaded my IPod with my collected music. As mentioned in my earlier post, I am yet to work on standardizing the mp3 tags so to structure and search my music better. But I thought I could listen to my entire collection. Another daunting problem surfaced, partly because of my swinging moods when listening to music. Consider the following playlist specifications that I would love to give:
  • Listen to A. R. Rehman's compositions
  • Listen to all Shahrukh Khan's songs
  • Listen to songs sung by Hari Haran and Shankar Mahadevan.
  • Listen to all songs of 1990's (1990-2000) time for old time's nostalgia.
  • Listen to duets (sung by a male and female singer)
  • Listen to songs sung by single artists
  • Listen to all telugu songs sung by Chitra
  • Listen to all tamil songs of Rajnikanth (yes, i listen to telugu, tamil, hindi and english songs)
  • and so on. But you get the general idea, right?
I listen to songs in RealPlayer, ITunes, Rhapsody and WinAmp, but I am not aware of having the option to get such queries executed for a different playlist according to my choice. And in iPod too there's no such option.

Such queries might be tough right now because the mp3 songs do not have the option to have all such information. For my queries to come to life, we need extra information such as:
singers tagged with (male/female), language of the song, actors/actresses in the song, composed by or music director and comprehensive genre. The genre field in mp3 tags usually takes it just as a single value i.e. movie, inspirational and movie, blues are considered two different genres. But songs of these genres are movie songs, so when there's a query that asks for just movie songs, songs belonging to both these genres should be collected in the playlist. Genre should ideally have a taxonomy, something like a broad category as to what type of music it is (movie, pop, rock, jazz, classical), followed by language in which it is sung (can take one or more values: english, hindi, telugu, tamil), followed by mood of the song (blues, tragedy, philosophical, inspirational and so on).

Currently, ID3 tags do not offer such flexibility. But can we store something in the 'Comments' field and make it possible to run such queries to generate dynamic playlists. Ofcourse, we also need a software that can read the 'Comments' field (or whatever we choose to) and filter out the songs that satisfy the query. I wonder why this requirement of dynamic playlists has not come up yet. Or did it and I am not aware of it? Let me know, while for a good solution to this, I mine.