Tuesday, August 12, 2014

ChefDK is easier to install Berkshelf than via gem on Amazon Linux AMI

I ended up installing the latest berkshelf, 3.x via gem on Amazon Linux AMI because it is quite difficult to deal with the dependency issues. Instead, I decided to use Chef Development Kit because it contains Berkshelf and is easy to install without worrying about such issues.

Installing berkshelf without specifying the version
But it failed to install because of the lack of the dependent libraries.
$ gem i berkshelf
Fetching: addressable-2.3.6.gem (100%)
Successfully installed addressable-2.3.6
Fetching: multipart-post-2.0.0.gem (100%)
Successfully installed multipart-post-2.0.0
Fetching: faraday-0.9.0.gem (100%)
Successfully installed faraday-0.9.0
Fetching: berkshelf-api-client-1.2.0.gem (100%)
Successfully installed berkshelf-api-client-1.2.0
Fetching: hashie-2.1.2.gem (100%)
Successfully installed hashie-2.1.2
Fetching: buff-extensions-1.0.0.gem (100%)
Successfully installed buff-extensions-1.0.0
Fetching: varia_model-0.4.0.gem (100%)
Successfully installed varia_model-0.4.0
Fetching: buff-config-1.0.1.gem (100%)
Successfully installed buff-config-1.0.1
Fetching: buff-ruby_engine-0.1.0.gem (100%)
Successfully installed buff-ruby_engine-0.1.0
Fetching: buff-shell_out-0.1.1.gem (100%)
Successfully installed buff-shell_out-0.1.1
Fetching: minitar-0.5.4.gem (100%)
Successfully installed minitar-0.5.4
Fetching: retryable-1.3.5.gem (100%)
Successfully installed retryable-1.3.5
Fetching: buff-ignore-1.1.1.gem (100%)
Successfully installed buff-ignore-1.1.1
Fetching: hitimes-1.2.2.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing berkshelf:
        ERROR: Failed to build gem native extension.

    /usr/bin/ruby2.0 extconf.rb
checking for clock_gettime() in -lrt... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib64
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/bin/ruby2.0
        --with-rtlib
        --without-rtlib
/usr/share/ruby/2.0/mkmf.rb:434:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
        from /usr/share/ruby/2.0/mkmf.rb:519:in `try_link0'
        from /usr/share/ruby/2.0/mkmf.rb:534:in `try_link'
        from /usr/share/ruby/2.0/mkmf.rb:720:in `try_func'
        from /usr/share/ruby/2.0/mkmf.rb:950:in `block in have_library'
        from /usr/share/ruby/2.0/mkmf.rb:895:in `block in checking_for'
        from /usr/share/ruby/2.0/mkmf.rb:340:in `block (2 levels) in postpone'
        from /usr/share/ruby/2.0/mkmf.rb:310:in `open'
        from /usr/share/ruby/2.0/mkmf.rb:340:in `block in postpone'
        from /usr/share/ruby/2.0/mkmf.rb:310:in `open'
        from /usr/share/ruby/2.0/mkmf.rb:336:in `postpone'
        from /usr/share/ruby/2.0/mkmf.rb:894:in `checking_for'
        from /usr/share/ruby/2.0/mkmf.rb:945:in `have_library'
        from extconf.rb:10:in `

'
Gem files will remain installed in /home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2 for inspection. Results logged to /home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/gem_make.out

Installing hitimes via gem
But it failed because the development tools should be installed before that.
$ gem i hitimes --verbose
HEAD https://rubygems.org/latest_specs.4.8.gz
302 Moved Temporarily
HEAD https://s3.amazonaws.com/production.s3.rubygems.org/latest_specs.4.8.gz
304 Not Modified
Installing gem hitimes-1.2.2
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/.travis.yml
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/CONTRIBUTING.md
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/HISTORY.md
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/LICENSE
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/Manifest.txt
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/README.md
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/Rakefile
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/examples/benchmarks.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/examples/stats.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/extconf.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_instant_clock_gettime.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_instant_osx.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_instant_windows.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_interval.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_interval.h
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_stats.c
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/hitimes_stats.h
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/java/src/hitimes/Hitimes.java
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/java/src/hitimes/HitimesInterval.java
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/java/src/hitimes/HitimesService.java
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/java/src/hitimes/HitimesStats.java
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/metric.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/mutexed_stats.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/paths.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/stats.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/timed_metric.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/timed_value_metric.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/value_metric.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/lib/hitimes/version.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/hitimes_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/interval_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/metric_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/mutex_stats_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/paths_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/spec_helper.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/stats_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/timed_metric_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/timed_value_metric_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/value_metric_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/spec/version_spec.rb
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/tasks/default.rake
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/tasks/extension.rake
/home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/tasks/this.rb
Building native extensions.  This could take a while...
/usr/bin/ruby2.0 extconf.rb
checking for clock_gettime() in -lrt... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib64
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/bin/ruby2.0
        --with-rtlib
        --without-rtlib
/usr/share/ruby/2.0/mkmf.rb:434:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
        from /usr/share/ruby/2.0/mkmf.rb:519:in `try_link0'
        from /usr/share/ruby/2.0/mkmf.rb:534:in `try_link'
        from /usr/share/ruby/2.0/mkmf.rb:720:in `try_func'
        from /usr/share/ruby/2.0/mkmf.rb:950:in `block in have_library'
        from /usr/share/ruby/2.0/mkmf.rb:895:in `block in checking_for'
        from /usr/share/ruby/2.0/mkmf.rb:340:in `block (2 levels) in postpone'
        from /usr/share/ruby/2.0/mkmf.rb:310:in `open'
        from /usr/share/ruby/2.0/mkmf.rb:340:in `block in postpone'
        from /usr/share/ruby/2.0/mkmf.rb:310:in `open'
        from /usr/share/ruby/2.0/mkmf.rb:336:in `postpone'
        from /usr/share/ruby/2.0/mkmf.rb:894:in `checking_for'
        from /usr/share/ruby/2.0/mkmf.rb:945:in `have_library'
        from extconf.rb:10:in `

'
ERROR:  Error installing hitimes:         ERROR: Failed to build gem native extension.     Building has failed. See above output for more information on the failure. Gem files will remain installed in /home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2 for inspection. Results logged to /home/ec2-user/.gem/ruby/2.0/gems/hitimes-1.2.2/ext/hitimes/c/gem_make.out

Installing "Development libraries" and "Development tools"
It succeeded without any fail.
$ sudo yum -y groupinstall "Development Libraries" "Development tools"
$ rpm -qa |grep ruby
ruby20-2.0.0.451-1.14.amzn1.x86_64
rubygem20-json-1.7.7-101.27.amzn1.x86_64
ruby20-libs-2.0.0.451-1.14.amzn1.x86_64
rubygem20-psych-2.0.0-1.14.amzn1.x86_64
rubygems20-2.0.14-1.14.amzn1.noarch
rubygem20-rdoc-4.0.1-2.18.amzn1.noarch
ruby20-devel-2.0.0.451-1.14.amzn1.x86_64
ruby20-irb-2.0.0.451-1.14.amzn1.noarch
ruby-2.0-0.3.amzn1.noarch
rubygem20-bigdecimal-1.2.0-1.14.amzn1.x86_64

Installing hitimes with gem again
It succeeded.
$ gem i hitimes --verbose
HEAD https://rubygems.org/latest_specs.4.8.gz
302 Moved Temporarily
HEAD https://s3.amazonaws.com/production.s3.rubygems.org/latest_specs.4.8.gz
304 Not Modified
Installing gem hitimes-1.2.2
...
Successfully installed hitimes-1.2.2
Parsing documentation for hitimes-1.2.2
Parsing sources...
100% [14/14]  lib/hitimes/version.rb                                           
Installing ri documentation for hitimes-1.2.2
Done installing documentation for hitimes after 0 seconds
1 gem installed

Installing berkshel via gem
It failed because libgecode failed to be compiled.
$ gem i berkshelf --verbose
...
virtual memory exhausted: Cannot allocate memory
make[1]: *** [gecode/int/extensional.o] Error 1
make[1]: *** Waiting for unfinished jobs....
fvirtual memory exhausted: Cannot allocate memory
make[1]: *** [gecode/int/rel.o] Error 1


make[1]: Leaving directory `/home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2/ext/libgecode3/vendor/gecode-3.7.3'
make: *** [compilelib] Error 2
extconf.rb:98:in `block in run': Failed to build gecode library. (GecodeBuild::BuildError)
        from extconf.rb:97:in `chdir'
        from extconf.rb:97:in `run'
        from extconf.rb:104:in `

'
ERROR:  Error installing berkshelf:         ERROR: Failed to build gem native extension.     Building has failed. See above output for more information on the failure. Gem files will remain installed in /home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2 for inspection. Results logged to /home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2/ext/libgecode3/gem_make.out

Installing libgecode with gem
It failed. So, I decided to use ChefDK instead.
$ gem i dep-selector-libgecode
...
virtual memory exhausted: Cannot allocate memory
make[1]: *** [gecode/int/element.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2/ext/libgecode3/vendor/gecode-3.7.3'
make: *** [compilelib] Error 2
extconf.rb:98:in `block in run': Failed to build gecode library. (GecodeBuild::BuildError)
        from extconf.rb:97:in `chdir'
        from extconf.rb:97:in `run'
        from extconf.rb:104:in `

'
Gem files will remain installed in /home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2 for inspection. Results logged to /home/ec2-user/.gem/ruby/2.0/gems/dep-selector-libgecode-1.0.2/ext/libgecode3/gem_make.out

Installing ChefDK followed by the official instruction
$ sudo yum install https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chefdk-0.2.0-2.el6.x86_64.rpm

Confirming the installation pathpath
$ ll `which chef`
lrwxrwxrwx 1 root root 20 Aug 11 10:38 /usr/bin/chef -> /opt/chefdk/bin/chef

Confirming Berkshelf installed
$ /opt/chefdk/embedded/bin/gem list | grep 'berkshelf'
berkshelf (3.1.3)
berkshelf-api (1.4.0)
berkshelf-api-client (1.2.0)

Verifying the main components of ChefDK
$ chef verify
Running verification for component 'berkshelf'
Running verification for component 'test-kitchen'
Running verification for component 'chef-client'
Running verification for component 'chef-dk'
..............
---------------------------------------------
Verification of component 'chef-dk' succeeded.
Verification of component 'berkshelf' succeeded.
Verification of component 'chef-client' succeeded.
Verification of component 'test-kitchen' succeeded.

Installing knife solo
Just make sure that you install knife solo because ChedDK does not contain it.
$ chef gem install knife-solo

ChefDK is quite helpful to resolve dependency issues.