Cassandra DSE, Testing Solr integration

SOLR support is one of the DataStax commercial offering for cassandra. which basically enable us to run a real-time solr query against the data in the cassandra. Here is one basic try of the features.

when you create the DSE cluster, you can change some node by setting the /etc/default/dse enable hadoop or Solr.
Change the SOLR_ENABLED or HADOOP_ENABLED one will add one more role to that Cassandra node.

For me, I have 8 Nodes as a whole cluster. 3 as regular cassandra, 3 as Hadoop node, 2 as SOLR, you can see fro the ops center view. or just through node tool.

Now Let’s create one KS called solrtest first through the OPscenter admin ui,

And create one column family called info using Cqlsh. and load some data.

DSE has some default mapping between the data stored in cassandra, and data needed to be indexed to SOLR.

By default, Nodes will be mapped to Shard. for my case, I have 2 Solr nodes, that means I have 2 Solr Shards. CF will be mapped to Core, so here I need to tell the system I want to index info column family.  for columns in that CF, will be mapped to solr field. we can pick which columns need to be indexed throught some level of configration called schema.xml which is the same file in SOLR.

in the /usr/share/dse-demo/wikipedia , there are some sample schemas and script.
  we change the schema.xml first, basically we just need two fields to be indexed, and default search field is comments,


then we need post his xml to SOLR, here is one script called 1-add-schema.sh


change the mapping url, shoul http://ANYSOLRNODE:8983/solr/resource/KEYSPACE.CF/solrconfig.xml

then we run the script to post the schema, and config file. (you are only need run this for one solr server, the solr server shared those configuration.)

after done, you can access.
http://ANYSOlrServer:8983/solr/KEYSPACE.CF/admin/ to see the solr admin ui,

when you run search, you can see the docs returned as expected.

if we change the query to comments:hello, then only first doc will be returned

If we insert more data, those data will be indexed on the fly.

search solr, 2 docs returned.

At the same time, you can use the CQL to query the data using solr syntax.

So that’s it. you may wondering what happened underneath when we run this query?

basically it will query all SOLR Nodes to run a distrubuted shared query, get the items key, and pull the data from cassandra.

R data mining IDE, rattle tutorial

Besides the Rcmdr IDE, Rattle is another one IDE for me to do interactive data minding or just data discovery.
installation is easy, just install rattle library, then load the library , and click yes to install the prerequisites components.

once done, run rattle() to load the Rattle IDE.

like the Rcmdr, you can load/import data. then you can explore the data. Always remember to click Execute to make it happen.
since it’s used for modeling also, you can specify which column is primary key(ident here,) which one is the response value here it is called Target.


then click explore tab, to do more exploring. like a summary.

click distribution, you can do some plotting.

in the correlation, you can tell which variable are correlated to others, blue means positive.

for the sample data, of course, revenue is correlated with request, impression and clicks.
has no relationship with fill_rate.
f

for the interactive , you can use plot builder to do a lot plotting. rember to install JDK and rJava library.

here is one cool bubble plot,

Also you can run transform, clustering.

R data minding package , Rcmdr tutorial/setup guide

being using the RStudio for a while , RStudio is a great IDE for R programmers, you need remember all the function/parameters. For the c# programmers, RStudio is just like the Visual Studio which has a great support for intellisense, IDE. project workspace, Version control. Rcmdr is a great Visual GUI tools for data mining, even for  just plotting/graphics. here is one basic tutorial,

Install Rcmdr, basically just click RStudio or Any R IDE, or just R command, try install package ‘Rcmdr’


once done, you can run library(Rcmdr) to startup the IDE, you may be prompted to install some components , just click yes to GO.

once done, you will see a new IDE, Rcmdr which has more menus.

basically, you can use the Data menu to import data from external data sources, or load data in packages, and run some data processing, like convert numbers to factors, or get a subset data
after data was loaded, you can view/edit the data, also from the script window, you can see the genrated R script,

Now, let’s run some basic plotting, select Graphics menu,->histogram, you will be asked to chose the varible to plot based on existing dataset.

Click OK, the graphics will be shown on IDE hosted the R, here will be Rstudio for me.

you can click statistics to create one Model like LInear model, or do some dimentional analysis

one you get the model, you can click models to visualize it.

Chrome Unknown connection to visicom-89.nationalnet.com

Just checked My chrome conenction, there someone are expected, someone are not. I saw 3 weird connection to Visicom-8x.nationalnet.com

then I ping the get the ip of those DNS, it’s 66.115.130.139, or 138


when you try access this page directly, you will see error.


weird, then I captured the Network traffic , oh got, saw the http traffic,

Follow it, it’s a http HEAD request

What the hell is the sherqsjgod host here?
after disable the extensions one by one ,it’s caused by the Web developer tools,


stupid plug-in! here is the official link,
https://chrome.google.com/webstore/detail/bfbameneiokkgbdmiekhjnmfkcnldhhm

Just disable it, the weird traffic will be gone.

How to : Load resource stream in WinRT / windows 8

Here is one quick tutorial showing you how to access embedded resource in winRT, which is much easier than windows phone 7.
Create one empty project Named App1, right click assets folder to create one text file, and right click the properties of the file, change the build action to embedded resource.


Now you can using the following code to query the available resource names.



please note, the GetTypeInfo method is one extension method, so you need put the System.Reflection in your using area.

similarly, you can using the code to get the stream reference, and read all the content.

How to: Fix windows 8 freezing issues for Apple Macbook / Pro

Just installed the RTM version of windows 8 on my Macbook using the bootcamp, just a plain OS. then installed the Chrome without any problem. After 10 to 15 minutes, the Computer freezes. then happened another 3 times randomly,what the hell?

I checked the event log, nothing special,


then searched Nvidea for beta drivers, it turns out there are one beta version driver for windows 8.


loaded , still no luck, then asked some microsoft friends, looks like another trip to disable the dynamictick, basically run as admin, and disable this option “bcdedit /set disabledynamictick yes”
If you don’t know how to run cmd in admin, run msconfig first,(Win+R to find the run utility first.)

in the tools tab, run the command prompt,


Now it has been 2 hours so far, no Crash luckily, hope it helps too.

How to: test Nginx with SPDY for .net application tutorial

Basically, Nginx has one patch which supports the SPDY, all you need is to download the dev version and apply the patch.
here is one step-by-step tutorial one how to install and config nginx to support SPDY In front, and any we app as the background.

1. Install OpenSSL 1.0.1, it's required because SPDY module uses
Next Protocol Negotiation TLS extension.
go to http://www.openssl.org/source/, pick up the 1.1 link. i.e http://www.openssl.org/source/openssl-1.0.1c.tar.gz

wget http://www.openssl.org/source/openssl-1.0.1c.tar.gz tar zxvf openssl-1.0.1c.tar.gz ls cd openssl-1.0.1c ./config make make install

2. Download nginx 1.3.x tar-gzip package, visit http://nginx.org/en/download.html,
or $ wget http://nginx.org/download/nginx-1.3.4.tar.gz
3. Unpack nginx-1.3 $ tar xvfz nginx-1.3.4.tar.gz $ cd nginx-1.3.4
4. Download and apply SPDY module patch from http://nginx.org/patches/spdy/
   $ wget http://nginx.org/patches/spdy/patch.spdy.txt
   $ patch -p0 < patch.spdy.txt
5. Configure nginx build
$ ./configure --with-http_ssl_module Use --with-openssl=/path/to/openssl-1.0.1, when building OpenSSL separately and statically linking.
Use --with-cc-opt and --with-ld-opt accordingly, if OpenSSL is installed as an optional library, e.g. on Mac OS X $ ./configure --with-http_ssl_module \ --with-cc-opt="-I/opt/local/include" \ --with-ld-opt="-L/opt/local/lib"
if you don’t have zlib, need install zlib-devel library
I don’t have the pcre library, so disable the urlrewrite features
./configure --with-http_ssl_module --with-openssl=/root/openssl-1.0.1c/ --without-http_rewrite_module
6. Build nginx
$ make
7. Install
make install.
Now we can test the hello world of nginx,by default, all bits are under /usr/local folder.
go to sbin, run nginx to test the version and configration optins,
then start the nginx
sbin/nginx Now you can access localhost (by default it listens on port) we can change the conf under conf/nginx.conf

Now, let’s chagnge the config to add a backend server, I will use a internal server with ip 192.168.209.1 and port 4444 as the example. no spdy so far,
Original backend server, just a startpage of a empty iis site,


let’s change the nginx conf to add this backend server.
basically we just change the location of the server to a proxy access,

location / { root html; index index.html index.htm; } change to. location / { proxy_pass http://192.168.209.1:4444/ ; proxy_redirect default; }

restart the nginx , when you access localhost again, the welcome nginx page will be replcaed as the IIS7 page.

Now, lets turn on the SPDY support.

Generate SSL key and certificate. because the Nginx SPDY module using the Next protocal negotiation tls extension. so we need using the openssl to generate a self signed Certificate. before we create and signed the certifiacte , we need first create one private key.

we can create the key under the nginx conf folder, the key is called localhost.key


once we get the private key, we will create one Certificate sign request using this key, the request will be stored as localhost.csr.

Now we can “sign” and generate the certificate.


the last step, when we create the private key, we specified a password, we can remove the password, since this key is used by the app instead of human.


So far , we get 4 files , two private keys(one without password), one cert request, and one final certificate. we will use the private key and final certificate.

then change the nginx.conf, final step

server { listen 80; server_name localhost; location / { proxy_pass http://192.168.209.1:4444/ ; proxy_redirect default; } After // I highlighted the difference. basically, put spdy and ssl key server { listen 443 ssl spdy; server_name localhost; ssl_certificate "localhost.crt"; ssl_certificate_key "localhost_nopass.key"; location / { proxy_pass http://192.168.209.1:4444/ ; proxy_redirect default; }

restart the service, and try https://localhost, you will see the spdy indicator is on. we are on SPDY mode.(search SPDY indicator In the chrome store.)


furthure moe, you can check chrome net internals,
go to chrome://net-internals/#events&q=type:SPDY_SESSION%20is:active and rehit the page, you will see session in spdy got captured here,