Online Retailers, 7 of 10 Didn’t protect their Customer information in a secure way/ Password reset page

From the error page design rules 101 , we should only tell the audience general error information instead of the detailed information which will tell something to hacks about your system. 
  
Now it’s holiday season, I get a chance to visit several E-commerce sites which I didn’t visit very often. then for some site, I forgot the password, luckily, every site has the password reset options. unfortunately, 7 of 10 sites didn’t secure the user information correctly. I will give some examples. good and bad ones. 

my of my favorite  sites, HpShopping.com.  when you input an email address , It will tell you immediately whether that email is a HPshooping customer or not. this process is not protected by any sort of CAPTCHA authentication , AND could be scripted as part of the spam campaign or targeted fishing. idea is very simple, go through the email db which has a lot junk addresses, and filter out those valid HPshooping customers, then send targeted fishing email.

bad example, HP shooping.


same thing for buy.com

staples.com, no luck,

Some good example.
Apple.com. when you input an email address whether it’s valid customer or not, just tell you a general information. it it’s true, you will the password reset notification.

that’s sweet design.  from the perspective of security, it’s secure. for user convenience. secure too. no extra steps required.

Also some good examples, that might cause some  inconvenience which need extract step to identity and key in the CAPTCHA code.

Amazon.com, for regular good user, they know whether the account is good or not immediately, but have to pass the CAPTCHA test. However, it’s SECURE.

Newegg has the similar approach. it combined Amazon and Apple design, which is the most secure one. It requires CAPTCHA and tells only general information.

How To: MySql replication between two clusters. capture data changes

Here , I just setup two VMs and following tutorial Installing and testing Mysql NDB Cluster on Centos Tutorial. On each Server, there is one NDB cluster with one API Node and two Data Nodes and one mgmt node.
  before we kick off the replication, first make sure the /etc/hosts file has the right entry for both Hosts. Otherwise you may get the error in mysql like no access.

101123  5:55:16 [Warning] IP address '192.168.30.22' could not be resolved: getnameinfo() returned error (code: -3). 101123  5:55:56 [Warning] IP address '192.168.30.22' could not be resolved: getnameinfo() returned error (code: -3).

Configuration changes on HOST Master.

On HostA mysql instance, create one user that will be used by slave server hostb to pull the replication information.

GRANT REPLICATION SLAVE ON *.* TO 'repluser'@'hostb' IDENTIFIED BY 'pass';

Chagne the /etc/my.cnf , add options to enable binlog logging on row level, and assign one server-id.

[mysqld] ndbcluster ndb-connectstring=localhost log-bin binlog-format=row server-id=1

backup the db using the mysql dump and copy to Slave server for the initial restore. (ALL DB objects will be covered in the mysqldump , like view , storeprocedue)

mysqldump –u root yourdatabase –master-data=1> ini.sql

Restart the Master mysqld daemon. that’s all the  changes.

Configuration changes on HOST Slave Server.

Assign one explicit server-id in the my.cnf

[mysqld] ndbcluster ndb-connectstring=localhost server-id=2

restore the db from the master backup

mysql –u root ourdatabase<init.sql

Change the master to the appropriate host.

CHANGE MASTER TO MASTER_HOST='hosta', MASTER_PORT=3306, MASTER_USER='repluser', MASTER_PASSWORD='pass';

start the slave service>.

slave start

 

Now any change will be synced to slave server.

if you have any problem to sync the data. always check the /usr/local/mysql/data/yourhostname.err

there are some system tables to maintain the binlog and replication status.

for example, in the master server mysqld which we enable the binarylog.
you can always query mysql.ndb_binlog_index.

 

here we have two log entries which represent two update in my case. If we want to replay the first one  in slave server. just stop the slave replication and change the master log file position.

here the log will be replayed and duplicate record will inserted into the table.

If we just want to capture Master change and push the change to other db store besides mysql.

we can change the slave server table engine to blackhole, which will never store any data.  then turn on the query logging.
now all changes on master data will be pushed to slave server , we can tell the change from the query log.

VMware player can’t open OVF file

When you get an OVF format VM image, you can’t open it with Vmware player directly. lucky there is a way to conver ovf VM image to VMX format.

Download the OVFTool from Vmware.
http://communities.vmware.com/community/vmtn/vsphere/automationtools/ovf

then run the ovftool to convert the image. here is the syntax

Ovftool c:\vm\myvm.ovf  c:\vm\vmxvm

How to : Installing and testing Mysql NDB Cluster on Centos Tutorial

AS I mentioned in How to Create a Mysql NDB cluster on a PC, windows 7 tutorial, It requires the same prerequisites to run a simple NDB cluster. I will go over a tutorial to create a one Mgmt node, Two storage Node One Api Node clusters in one single Centos instance.

Download the Linux version of Mysql cluster community edition which is free to use . 
Unzip it and copy the folder to /usr/local/mysql, here you may copy to a folder like /usr/local/mysql_xxx where xxx is the edition, then put a link named mysql to that folder.

scripts, tar -zxvf mysql-cluster-gpl-7.1.8-linux-x86_64-glibc23.tar.gz #unzip it cp -r mysql-cluster-gpl-7.1.8-linux-x86_64-glibc23 /usr/local/ ln -s /usr/local/mysql-cluster-gpl-7.1.8-linux-x86_64-glibc23/  /usr/local/mysql

Create a user named mysql.

useradd mysql

setup the Mysql service.

cd /usr/local/mysql # you must be in this folder to run the install script ./scripts/mysql_install_db once done, copy the demon script cp support-files/mysql.server  /etc/init.d/mysql chown -R mysql:mysql /usr/local/mysql/data

Now you can run service  mysql start to startup a regular mysql service.

Time to create a NDB config ini, there are some templates file like config-xxx.ini in the support-files directory, just copy and paste. and change the nodes per you requirement.
I will put a config file in /ndb/config.ini
and put all NDB data into /ndb/folder

mkdir /ndb cp /usr/local/mysql/support-files/config.medium.ini /ndb/config.ini

then change the config.ini just created as the following content.

[NDBD DEFAULT] NoOfReplicas: 2 DataDir: /ndb [MGM DEFAULT] PortNumber: 1186 DataDir: /ndb [NDB_MGMD] Id: 1 [NDBD] Id: 2 [NDBD] Id: 3 [API] Id: 4

after saving the chagnes to /ndb/config.init, run a commant ndb_mgmtd –f /ndb/config.ini –-ini to initialize the ndb folder

cd /usr/local/mysql/bin ./ndb_mgmd -f /ndb/config.ini  --init ./ndb_mgm

Only mgmt node is active in this cluster now.

run ./ndb twice to start Two  NDB Data Nodes.

run the show in ndb_mgm to refersh the status. Two data ndoes are back.
 

one more step left, change the mysqld API Node to point to ndb cluster ( ask it to pickup the ndb engine and ndb node configuration.  all it need is the connectiong to ndb mgm node, node will push all information to API nodes.)

Create one file in /etc/my.cnf as content below.

[mysqld] ndbcluster ndb-connectstring=localhost

restart the mysql service. API node is On now

let’s run a helloworld test.

Now you can run several utility to make sure the data is distributed to two NDB data nodes evenly.

ndb_desc –d hello foo -p

here two partitions, one has two records, one has one record.

run ndb_mgm, all report memory

How To : installing and testing greenplum single node edition on Centos 5.5

Download the single node edition, Here I use the version 4.0 for redhat /centos , greenplum-db-4.0.0.4-build-1-RHEL5-x86_64.bin

check OS version,

[root@localhost Desktop]# cat /etc/redhat-release CentOS release 5.5 (Final)

if you run the installer on unmatched version of centos, will get an error, “Installer will only install on RedHat/CentOS x86_64”
here run Uname –m, It will tell the hardware spec, 386 or x86_64.

[root@localhost ~]# uname -m x86_64

run the installer directly.

[root@localhost Desktop]# ./greenplum-db-4.0.0.4-build-1-RHEL5-x86_64.bin make sure you have the execute permission. If not, run chmod 722 greenplum-db-4.0.0.4-build-1-RHEL5-x86_64.bin

read and accept the agreement,
by default the bits will be put into folder /usr/local/greenplum-version.  Enter yes to accept the default settings. like folder path, once done. you will get the message like

source /usr/local/greenplum-db-4.0.0.4/greenplum_path.sh

Create a dedicated user to run the process , and folders to hold segments and maser data. given user the folder access permission.

Here we will create a user named gp Create a maser folder /data/master Create 4 segment folders /data/seg1, /data/seg2,/data/seg3/,/data/seg4 Change ownership folder of /data/* to gp

 
Copy one instance from the sample config and change the setting like folder path , listen port.

cd /usr/local/greenplum-db-4.0.0.4/docs/cli_help [root@localhost cli_help]# cp gp_init_singlenode_example /home/gp/ [root@localhost cli_help]# cp single_hostlist_example /home/gp/ [root@localhost cli_help]#

Switch to user gp, go to home foder, then change two files we just copied here.

gp_init_singlenode_example content change. MACHINE_LIST_FILE=./single_hostlist_example declare -a DATA_DIRECTORY=(/data/seg1 /data/seg2 /data/seg3 /data/seg4) #four segment folders MASTER_DIRECTORY=/data/master #master diferctory

Initialize the system. (create folders and start up several postgres processes.)

[gp@localhost ~]$ gpssh-exkeys -f single_hostlist_example gpinitsystem -c gp_init_singlenode_example

once done, when you run ps -aux|grep /data/, you will see 5 postgresq processes here, one master , four segment instances.

Now the single Node cluster is ready, we can use the psql to connect to the instance and have some fun.

Create one test database CreateDB androidtest psql (8.2.14) Type "help" for help. androidtestdb=# Create table public.test( id int primary key, state varchar(30) not null); NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "test_pkey" for table "test" CREATE TABLE androidtestdb=# \d+ public.test;                    Table "public.test" Column |         Type          | Modifiers | Description --------+-----------------------+-----------+------------- id     | integer               | not null  | state  | character varying(30) | not null  | Indexes:     "test_pkey" PRIMARY KEY, btree (id) Has OIDs: no Distributed by: (id) Now we just created one table test which is disturbed by id. which means if we put more data, the data will be distributed and dispatched to different segments by column iD.  let’s run a simple sql. insert into public.test values (1,'AL') ,(2,'AK') ,(3,'AS') ,(4,'AZ') ,(5,'AR') ,(6,'CA') ,(7,'CO') ,(8,'CT') ,(9,'DE') ,(10,'DC') ,(11,'FM') ,(12,'FL') ,(13,'GA') ,(14,'GU') ,(15,'HI') ,(16,'ID') ,(17,'IL') ,(18,'IN') ,(19,'IA') ,(20,'KS') ,(21,'KY') ,(22,'LA') ,(23,'ME') ,(24,'MH') ,(25,'MD') ,(26,'MA') ,(27,'MI') ,(28,'MN') ,(29,'MS') ,(30,'MO') ,(31,'MT') ,(32,'NE') ,(33,'NV') ,(34,'nh') ,(35,'NJ') ,(36,'NM') ,(37,'NY') ,(38,'NC') ,(39,'ND') ,(40,'MP') ,(41,'OH') ,(42,'OK') ,(43,'OR') ,(44,'PW') ,(45,'PA') ,(46,'PR') ,(47,'RI') ,(48,'SC') ,(49,'SD') ,(50,'TN') ,(51,'TX') ,(52,'UT') ,(53,'VT') ,(54,'VI') ,(55,'VA') ,(56,'WA') ,(57,'WV') ,(58,'WI') ,(59,'WY') ; INSERT 0 59 How can we tell whether the data are distributed across segments evenly. in each table, there is one system column called gp_segment_id select gp_segment_id, id, state from public.test; gp_segment_id | id | state ---------------+----+-------              3 |  2 | AK              3 |  6 | CA              3 | 10 | DC              3 | 14 | GU              3 | 18 | IN              3 | 22 | LA              3 | 26 | MA you can run a basic group query androidtestdb=# select gp_segment_id, count(*) from public.test group by gp_segment_id; gp_segment_id | count ---------------+-------              3 |    15              0 |    15              1 |    14              2 |    15    data is evenly distributed across four segments. also you can run a analyze to tell the execution plan. which will also tell you the segments scan are handled in a parallel way. androidtestdb=# EXPLAIN  select * from public.test;                                  QUERY PLAN -----------------------------------------------------------------------------  Gather Motion 4:1  (slice1; segments: 4)  (cost=0.00..4.59 rows=15 width=7)    ->  Seq Scan on test  (cost=0.00..4.59 rows=15 width=7) (2 rows) EXPLAIN analyze  select * from public.test;                                                    QUERY PLAN                              ---------------------------------------------------------------------------------------------------------------- Gather Motion 4:1  (slice1; segments: 4)  (cost=0.00..4.59 rows=15 width=7)    Rows out:  59 rows at destination with 0.515 ms to first row, 1.215 ms to end.    ->  Seq Scan on test  (cost=0.00..4.59 rows=15 width=7)          Rows out:  Avg 14.8 rows x 4 workers.  Max 15 rows (seg0) with 0.026 ms to first row, 0.029 ms to end. Slice statistics:    (slice0)    Executor memory: 139K bytes.    (slice1)    Executor memory: 155K bytes avg x 4 workers, 155K bytes max (seg0). Total runtime: 2.038 ms (8 rows)

Windows: Start or Stop a Service on remote server using wmic

WMIC has been a built-in command since XP as a interface to WMI operations. Here is a basic task , stop or start the windows service on remote server.

Run WMIC on command prompt, or Run box.


By default, the WMIC connect to local WMI service. If you want to connect to a remote Node.
put

/Node:yourremote Server /User:account enter you password.

When you run /? , you will be able to see all the built-in Objects that you can interact with.

Service List, it will show you all services installed on the server

If you want to list only those service a running, and list the name caption only (like the sql projection syntax. )

service where state=”running” get name, caption, status

get a specified service status, like iisadmin

If you want to control the service, like stop the service or restart the service. 

to list all the operation on the service ,
Service where name=”iisadmin” call

the same approach applies to the process management.

process where name=”notepad.exe” call terminate close the notepad.exe process to shutdown the server terminate the winlogon.exe

Also you can use PSexec to run process on the remote server directly. http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

yum updates through Microsoft ISA proxy which needs NTLM authentication /NTLM Authorization Proxy Server HTTP Error 407: Proxy Authentication Required

If you have a Centos VM or any other VM that need access internet to update or download some bits. And the Host machine is behind the ISA proxy which is NTLM authentication enabled.  Here is the basic Idea to make the guest OS can access the internet without having the Authentication problem.

for some security reason, IT admin might disabled the Basic authentication for ISA access. If you use YUM and have to pass the proxy. Yum only support Basic authentication. so there is the problem. if you put one ISA proxy into /etc/yum.conf like proxy=http://youproxyserver:port and run yum update, you will get 407 error which means authentication failed. even you put the username password into the config. (why? proxy server only accept ntlm authentication)

login as: root root@192.168.209.128's password: Last login: Fri Nov  5 02:25:34 2010 [root@linuxserver ~]# vi /etc/yum.conf [root@linuxserver ~]# cls [root@linuxserver ~]# yum update kernel Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: centos-distro.cavecreek.net * updates: centos.eecs.wsu.edu * addons: mirror.nyi.net * extras: centos.promopeddler.com http://centos-distro.cavecreek.net/centos/5.5/os/i386/repodata/repomd.xml: [Errno 14] HTTP Error 407: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied.  )

answer, install one NTLM Authorization Proxy Server on Host or guest OS. (the  APS is python based, run anywhere that has python.) 'NTLM Authorization Proxy Server' (APS) is a proxy software that allows you to authenticate via an MS Proxy Server using the proprietary NTLM protocol. then change the yum.config to use the new proxy server which is anonymous.

more steps,
Download and Install Python
Download and install NTLM Authorization Proxy Server

Change the server.cfg,  here is the key change

PARENT_PROXY:ISA01 PARENT_PROXY_PORT:8080 Accept:text/html, application/xhtml+xml, */* User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; MS-RTC LM 8) NT_DOMAIN:test.com USER:useraccount PASSWORD:password LM_PART:1 NT_PART:1 NTLM_FLAGS: 07820000

Then startup the proxy utility ntlmaps-0.9.9.0.1\runserver.bat

Now change the yum.conf to the new proxy ISA01, Yum works now.

for Other http client like Wget, you need Export http_proxy=http://yourproxy:port

 

reference,
http://www.centos.org/docs/5/html/yum/sn-yum-proxy-server.html

CustomBinding binaryMessageEncoding error, Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service.

this is an Issue about the WCF Bindings. By default, if you use wshttpbinding or httpbinding, The http channel will use the Text based encoding which is not very efficient in terms of the network transfer.  however, you can combine one custombing, just pick up the binaryencoding which is been used by tcpbinding, and httptransfer as the transport channel.

you can create one customBinding like this,

<customBinding>        <binding name="csbinding">          <binaryMessageEncoding>          </binaryMessageEncoding>          <httpTransport>          </httpTransport>        </binding>      </customBinding>

if the service is hosted in IIS, and the anonymous access for this folder has been turned off.  you may get the follow error when you try to access the svc. even just the wsdl access.

erver Error in '/DemoWCF' Application. Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.NotSupportedException: Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service. Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. Stack Trace: [NotSupportedException: Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service.] System.ServiceModel.Channels.HttpChannelListener.ApplyHostedContext(VirtualPathExtension virtualPathExtension, Boolean isMetadataListener) +11449553 System.ServiceModel.Activation.VirtualPathExtension.ApplyHostedContext(TransportChannelListener listener, BindingContext context) +75 System.ServiceModel.Channels.HttpTransportBindingElement.BuildChannelListener(BindingContext context) +119 System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener() +66 System.ServiceModel.Channels.MessageEncodingBindingElement.InternalBuildChannelListener(BindingContext context) +67 System.ServiceModel.Channels.BinaryMessageEncodingBindingElement.BuildChannelListener(BindingContext context) +46 System.ServiceModel.Channels.BindingContext.BuildInnerChannelListener() +66 System.ServiceModel.Channels.Binding.BuildChannelListener(Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, BindingParameterCollection parameters) +124 System.ServiceModel.Description.DispatcherBuilder.MaybeCreateListener(Boolean actuallyCreate, Type[] supportedChannels, Binding binding, BindingParameterCollection parameters, Uri listenUriBaseAddress, String listenUriRelativeAddress, ListenUriMode listenUriMode, ServiceThrottle throttle, IChannelListener& result, Boolean supportContextSession) +337 System.ServiceModel.Description.DispatcherBuilder.BuildChannelListener(StuffPerListenUriInfo stuff, ServiceHostBase serviceHost, Uri listenUri, ListenUriMode listenUriMode, Boolean supportContextSession, IChannelListener& result) +647 System.ServiceModel.Description.DispatcherBuilder.InitializeServiceHost(ServiceDescription description, ServiceHostBase serviceHost) +1364 System.ServiceModel.ServiceHostBase.InitializeRuntime() +37 System.ServiceModel.ServiceHostBase.OnBeginOpen() +27 System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout) +49 System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) +261 System.ServiceModel.HostingManager.ActivateService(String normalizedVirtualPath) +121 System.ServiceModel.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath) +479 [ServiceActivationException: The service '/DemoWCF/Service.svc' cannot be activated due to an exception during compilation. The exception message is: Security settings for this service require 'Anonymous' Authentication but it is not enabled for the IIS application that hosts this service..] System.ServiceModel.AsyncResult.End(IAsyncResult result) +11527290 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result) +194 System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(HttpApplication context, Boolean flowContext) +176 System.ServiceModel.Activation.HttpModule.ProcessRequest(Object sender, EventArgs e) +278 System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +68 System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75

for the fix. just right click the .svc file in IIS and enable the anonymous access.

Tutorial , Firebug , Free Commercial IE httpwatch alternative

HttpWatch is a great Commercial Tool to capture the Http traffic, enable admin to do the user end http analysis. like check the Cache is correct, client did get the Gzip version response. If you are looking for a free alternative.  FireBug for fixfox is my top choice.  Here is a basic tutorial one Firebug.

Install Firebug on Firefox.
Click tools->add-ons, search firebug. click to install the add-on and restart firefox.

After loaded the firebug, Open Firefox, press F12 or click add-ons to turn it on.

View the Http Request /Response Header. Click the dropdown arrow to enable the traffic capturing.

When you access Wikipedia mainpage, http://en.wikipedia.org/wiki/Main_Page, you will see the resource get requested to web servers. and the timeline.

click one url , i.e the first one. you can tell the server is apache, the wiki used Cache heavily. here the cache could be squid or varnish.  the we do get the gzip version response.

 
scroll down, you can see the request header the firefox is sending out.

If you press Shift and click the refersh in firefox. you will see that the firefox tell the edge server to reload th request and never pickup the cached version.

in the HTML tab, click the pointer to a object like the logo here., you can see the img url and css styles.