How to increase Elasticsearch Shard recovery Speed

To Increase Shard recovery speed do this:


Increase node recovery speed

PUT http://es:9200/_cluster/settings

{
"persistent" : {
"indices.recovery.max_bytes_per_sec": "200mb",
"indices.recovery.max_concurrent_file_chunks": 5,
"cluster.routing.allocation.node_concurrent_recoveries" : 5
}
}

Links:

https://www.elastic.co/guide/en/elasticsearch/reference/current/shards-allocation.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/recovery.html

Shell Script to SSH into servers

This is a simple script that can ssh into a server by typing the password for you.

Filename: login

#!/usr/bin/expect
eval spawn ssh username@servername
expect "assword:"
send "server_password\r"
interact

# to run the script
# ./login

In this below script you can pass arguments to the shell script
Filename: login

#!/usr/bin/expect
set num [lindex $argv 0];
eval spawn ssh username@servername-$num
expect "assword:"
send "server_password\r"
interact

# make sure to give executable permission to script.

chmod +x login

# to run the script and login to server-01.

./login 01

Now add the script path to your .bashrc file as alias, so that you can access this script from any path.

# vi ~/.bashrc

alias login='/path/to/script/file/login'

After updating the .bashrc file, you will have to source.

source ~/.bashrc

Now call the script from anywhere in your shell.

login 01

Enjoy!!!

How to install Ruby 2.X and Rails 5.X using RVM

The best way to install Ruby on Rails is using RVM in Linux.

Advantages of RVM,
1. You can shift between multiple versions of Ruby easily.
2. You don't have to worry about install Ruby dependencies. RVM install will take care of installing them. Some of the ruby dependencies are : gawk, autoconf, automake, bison, libffi-dev, libgdbm-dev, libncurses5-dev, libsqlite3-dev, libtool, libyaml-dev, pkg-config, sqlite3, zlib1g-dev, libgmp-dev, libreadline6-dev, libssl-dev
3. Easily upgrade Ruby version and try out a latest version to make sure your project works with the latest Ruby. If something breaks, changing the version is easy.
4. Delete unwanted Ruby version anytime :)

Installation steps.

Step 1. Install RVM

Source : https://rvm.io/rvm/install

$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

\curl -sSL https://get.rvm.io | bash -s stable
$ source /etc/profile
$ source ~/.rvm/scripts/rvm
$ type rvm | head -n 1  (this command returns 'rvm is a function' )

Step 2. Install Ruby

To List all known Ruby versions by rvm

$ rvm list known

From the list, pick a version and install ruby

$ rvm install ruby-2.3.7

Step 3. Install Rails

Rails is a Ruby gem. I always install bundler gem along with rails installation because it will be used in all Rails application to install dependencies

$ gem install bundler
$ gem install rails

Step 4. check version and start a project

# To check the Ruby version
$ ruby -v

# to check the Rails version

$ rails -v

# to start a new rails project with Mysql database. Rails by default uses sqlite. If you want to use any other database you will have to specify the adapter in config/database.yml

$ rails new project_name -d mysql

# Go inside the new project 

$ cd project_name

# to install the dependencies

$ bundle install

# To start the server

$ rails s

Now in the browser you can access the rails starter web page using,

http://localhost:3000

Getting started with Python Flask application with MVC structure

I have added a MVC structure and getting started with Python flask application in my Github. 
I will continue to contribute to this repository with Database connections and JWT Authentication.

Lets get started with Python and Flask :)

Here is the Github link for my project.

https://github.com/srikanthjeeva/python-flask-getting-started

Structure your Flask application like a Rails Application.

I'm a Rails developer. I like the way a Rails application is structured and I wanted to implement the similar kind of structure for a Python Flask web Application. Flask by default will not give you this structure. So make use of the getting started with Python flask repository above. The Code structure will look like this:

python-flask-getting-started/
|-- app
| |-- config.py
| |-- controllers
| | |-- hello_controller.py
| | |-- __init__.py
| |-- helpers
| | |-- common_helpers.py
| | |-- __init__.py
| |-- __init__.py
| |-- models
| | |-- __init__.py
| `-- templates
| |-- hello.html
| `-- __init__.py
|-- log
| `-- test.log
|-- README.md
|-- requirements.txt
`-- server.py

Prerequisite

Make sure you have installed python :) To check python version,

python -V

Make sure you have installed python-pip. If you are using Ubuntu, you can apt-get like this

sudo apt-get install python-pip

To use the Python Flask Starter template.

Step 1 : Checkout code

git clone https://github.com/srikanthjeeva/python-flask-getting-started.git cd python-flask-getting-started

Step 2: Install dependencies

(use sudo if required)

pip install -r requirements.txt

Step 3 : Start the server

python server.py

In browser go to,

http://localhost:5000

or

http://localhost:5000/hello

or for UI

http://localhost:5000/ui/hello

Converting JSON to LCOV (infofile)

Wrote a small NPM module that converts JSON to LCOV data

https://github.com/srikanthjeeva/hitmap_json_to_lcov

Install Cpanm module error : No such file or directory opening compressed index

I got this error while installing Rest client perl module

$ HOME=/tmp /home/perl/5.10/bin/cpanm REST::Client
! Finding REST::Client on cpanmetadb failed.
! cannot open file '/tmp/.cpanm/sources/http%www.cpan.org/02packages.details.txt.gz': No such file or directory opening compressed index
! Couldn't find module or a distribution REST::Client

Solution:

The problem is because of "LWP::Protocol::https" module. Removing the directory worked for me.

$ cd /perl_installed_path/perl/5.10/lib/site_perl/5.10.1
$ rm –rf LWP*

or try with option "--no-lwp"

cpanm REST::Client --no-lwp

Error while installing Perl 5.10.1

a -Wdeclaration-after-statement -Wendif-labels -Wc++-compat

cc -fstack-protector -L/usr/local/lib -o miniperl \

gv.o toke.o perly.o pad.o regcomp.o dump.o util.o mg.o reentr.o mro.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o universal.o xsutils.o globals.o perlio.o perlapi.o numeric.o mathoms.o locale.o pp_pack.o pp_sort.o \

miniperlmain.o opmini.o perlmini.o

pp.o: In function `Perl_pp_pow':
pp.c:(.text+0x2d79): undefined reference to `pow'
pp.o: In function `Perl_pp_modulo':
pp.c:(.text+0x3b22): undefined reference to `floor'
pp.c:(.text+0x3b58): undefined reference to `floor'
pp.c:(.text+0x3b90): undefined reference to `fmod'
pp.o: In function `Perl_pp_atan2':
pp.c:(.text+0x8985): undefined reference to `atan2'
pp.o: In function `Perl_pp_sin':
pp.c:(.text+0x8b22): undefined reference to `sin'
pp.o: In function `Perl_pp_int':
pp.c:(.text+0x8fc1): undefined reference to `floor'
pp.c:(.text+0x9031): undefined reference to `ceil'
pp.o:(.rodata+0x120): undefined reference to `cos'
pp.o:(.rodata+0x128): undefined reference to `sin'
pp.o:(.rodata+0x130): undefined reference to `sin'
pp.o:(.rodata+0x138): undefined reference to `exp'
pp.o:(.rodata+0x140): undefined reference to `log'
pp.o:(.rodata+0x148): undefined reference to `sqrt'
pp_pack.o: In function `S_pack_rec':
pp_pack.c:(.text+0x740b): undefined reference to `floor'
pp_pack.c:(.text+0x742e): undefined reference to `floor'
pp_pack.c:(.text+0x7453): undefined reference to `floor'
collect2: error: ld returned 1 exit status
makefile:347: recipe for target 'miniperl' failed
make: *** [miniperl] Error 1

This link helped me to find what i was missing: https://rt.perl.org/Public/Bug/Display.html?id=55980 and finally this worked for me

Step 1: find your lib path which has all libm* files

In most systems it will be under /usr/lib64. Search for the files in your system.

system:/usr/lib64 $ ls libm*
libm.a libmcheck.a libmenuw.so.5 libmpc.so.3.0.0 libmpfr.so.4 libmpx.so.0.0.0 libm.so libmvec.a libmagic.so.1 libmenu.so.5 libmenuw.so.5.9 libmpdec.so.2 libmpfr.so.4.1.4 libmpxwrappers.so.0 libmspack.so.0 libmvec_nonshared.a libmagic.so.1.0.0 libmenu.so.5.9 libmpc.so.3 libmpdec.so.2.4.2 libmpx.so.0 libmpxwrappers.so.0.0.0 libmspack.so.0.1.0 libmvec.so

Step 2: Use the path with glibpth option while doing configure

./Configure -des -Dglibpth='/lib/lib64 /usr/lib64'

or I see some forums suggested plibpth

./Configure -des -Dplibpth='/lib/lib64 /usr/lib64' -Dglibpth='/lib/lib64 /usr/lib64' -Dlibpth='/lib/lib64 /usr/lib64'

I have answered the same here @Stackoverflow: http://stackoverflow.com/a/43878868/453486