Wednesday, September 29, 2010

Test Windows Phone 7 built-in apps in your simulator

By default, when you install the developer tools from developer.windowsphone.com, there is no other Built-in applications except on Internet Explorer application.

If you want to experience the full set like the following simulator.

image

image

All you need is download a customized image from http://forum.xda-developers.com/showthread.php?t=784523

or simply search WM70C1.en-US.bin

then copy the images to C:\Program Files (x86)\Microsoft SDKs\Windows Phone\v7.0\Emulation\Images

then when you click run in visual studio, you will get the image as shown above.

Windows Phone 7 great features that Iphone /Android doesn’t have

Windows phone features

  • Developer tools + popular programming language.( Silverlight, Visual studio , Team suite. Powerful and free )
    • Blend for designers
    • Visual studio for developers, XNA and Silverlight
    • Team integration, like bug control. Reporting, CMMI-compliance
    • Iphone use the Objective-C, not popular as C# and java
    • Android use java, but no Designer tools , eclipse is just a basic IDE. Blend based , great behavior, animation designer support
    • clip_image001
  • Define one minimum hardware spec for all phones(like With GPS, at least 5M pixels camera. . Instead of just one hardware IPhone(No Keyboard)
  • you define the important things or Life style for you. ( if you think email is important, drag it to the top of home screen. The app icon shows you how many emails you get. Also you can drag gmail, hotmail, yahoomail Icons instead of just one email)
    • Pin a people like you friend or boss to the homescreen
  • Exchange, keep the server policy in sync on phone. ( Security, audit, compliance)
  • Panorama experience, ( most phone just scroll up and down, phone like a full-range length. You can scroll left and right. ), Like People as a contact, scroll right to show the what's new in social network. Email conversations
  • Navigation patterns, Three navigation button(go back, home and search)( go back , Android has the same features+menu button.)
    • Application switching.
    • clip_image002
  • Phone as a camera
    • No need to start the application by clicking several menus.
    • Just click the camera button even in standby mode
    • Both Iphone and android need you start the application explicitly.
    • Slide left and right to preview the pics you taken
  • Xbox Live
    • Gaming experience. Synergy integration like you avatar
    • You achievement
  • Office integration.
    • PPT, excel,
    • Sharepoint.
    • View and doc and update it , then upload it to sharepoint.
  • Features have been in other phones
    • Voice search like iphone and android
      • Click and home windows button to turn on the voice search.
    • Declare what permission you app need
      • Like access internet, call phone. Etc.
      • clip_image003
    • Android
      • clip_image004
  • Trial API
    • One line of code
  • Advertising SDK for windows phone 7
    • Download from ms download center

Thursday, September 23, 2010

How to download and compile, run Tika on windows tutorial

You might get scarred when you first try to download and run Tika on windows, If you dont have some experience of SVN and Maven like me. Here is a quick tutorial to go through these processes.

1. Use subversion client to download the source code of tika.

there is one Windows Shell Extension for Subversion, just download and install it to your windows box. then Right click one folder like C:\temp\tike, and CLick the Svn checkout context menu.

image

enter the SVN source url. http://svn.apache.org/repos/asf/tika/trunk

image

it may take couple seconds to download the source code . Click Ok when done.

image

2. Download Maven , the Build utility like the msbuild, ant. and put the mvn.bat folder to windows PATH.

After the path is set, you should be able to run “mvn” at the command prompt.

image

3. Go the the download tike source folder c:\temp\tika. and run “mvm install”

the builder will download necessary component and compile the project. this make take a while

image

4. run the tika app now.

go to that folder, run “java –jar tika-app-0.8-snapshot.jar –m a.txt”

to pull the metadata of a.txt

image

or –t  yourpdf.pdf to extract the pdf file content

Wednesday, September 22, 2010

tomcat 7 with solr 1.4, HTTP Status 404 - /solr/admin/file/index.jsp

there might be something wrong between the solr 1.4 and tomcat 7. here is one quick story.

After I install the solr 1.4 in tomcat 7. I Open the  /sor/admin and click schema or config .

image

get 404 error.

image

the answer here is pretty simple after I did a lot debugging. Just remove the last slash.

new url will be http://localhost:8888/solr/admin/file?file=schema.xml
   instead of http://localhost:8888/solr/admin/file/?file=schema.xml

after this, you can change the index.jsp locatd in webapps/solr/admin/index.jsp

<td>
  <% if (null != core.getSchemaResource()) { %>
  [<a href="file?file=<%=core.getSchemaResource()%>">Schema</a>]
  <% }
     if (null != core.getConfigResource()) { %>
  [<a href="file?file=<%=core.getConfigResource()%>">Config</a>]
  <% } %>

Solr CSharp /.NET Client Solrsharp , exception Object reference not set to an instance of an object.

After we installed the Solr 1.4 , we use use the solrj as the library Client integration with Solr by using SolrJ .

the latest version of solr is 1.4.x, however, if you want to use the C# client SolrSharp which has not been update sine late 2007. you might need to change something to make the client works with the new version of Solr. say 1.4.

the first exception is called Object reference not set to an instance of an object.

here is one basic snippet to upload one document to solr server via solrsharp. in this example, My solr server is listening on 9999 port. and I started the tcptrace which listens on 8888 and forward all traffic to localhost 9999. based on this, we can see the raw traffic between client and solr server.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using org.apache.solr.SolrSharp.Configuration;
using org.apache.solr.SolrSharp.Update;
using org.apache.solr.SolrSharp.Indexing;

namespace ConsoleApplication6
{
    class TestSolrSharp
    {
        public static void Main()
        {
            //upload document
            try
            {
                SolrSearcher solrSearcher = new SolrSearcher("http://localhost:8888/solr", Mode.ReadWrite);
                SolrUpdater update = new SolrUpdater(solrSearcher);

                UpdateIndexDocument doc = new UpdateIndexDocument();
                doc.Add("id", "1");
                doc.Add("title", "test solr");

                update.PostToIndex(doc, true);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }
        }
    }
}

When I run it , I get the following error.

System.NullReferenceException: Object reference not set to an instance of an object.
   at org.apache.solr.SolrSharp.Configuration.Schema.SolrSchema..ctor(SolrSearcher solrSearcher) in C:\Users\xx\Downloads\solrsharp-Dec-30-2007\src\Configuration\Schema\SolrSchema.cs:line 76
   at org.apache.solr.SolrSharp.Configuration.SolrSearcher.SetSolrSchema() in C:\Users\xx\Downloads\solrshar
p-Dec-30-2007\src\Configuration\SolrSearcher.cs:line 93
   at org.apache.solr.SolrSharp.Configuration.SolrSearcher..ctor(String solrUrl, Mode searcherMode) in C:\Users\xx\Downloads\solrsharp-Dec-30-2007\src\Configuration\SolrSearcher.cs:line 77
   at ConsoleApplication6.TestSolrSharp.Main() in C:\Users\xx\documents\visual studio 2010\Projects\ConsoleA
pplication6\ConsoleApplication6\Class1.cs:line 18
Press any key to continue . . .


then I check the tcptrace traffic, get the access exception.

image

it looks like the client library will first pull the Scheme.xml of solr. When it try to get the schema.xml from /solr/admin/get-file.jsp?file=schema.xml, the solr server returns error of No Access.

So we need to make sure http://localhost:9999/solr/admin/get-file.jsp?file=schema.xml is accessible. Why we dont’t have access at this time.

Let’s check the soure code of the get-file.jsp, as shown in the bellow.

<%@ page contentType="text/plain; charset=utf-8" pageEncoding="UTF-8" %>
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<%@ page import="org.apache.solr.core.Config,
                 org.apache.solr.core.SolrCore,
                 org.apache.solr.core.SolrConfig,
                 java.io.InputStream,
                 java.io.InputStreamReader,
                 java.io.Reader,
                 java.util.StringTokenizer,
                 java.util.logging.Logger"%>
<%!
  static Logger log = Logger.getLogger(SolrCore.class.getName());
%>
<%
  // NOTE -- this file will be removed in a future release
  log.warning("Using deprecated JSP: " + request.getRequestURL().append("?").append(request.getQueryString()) + " -- check the ShowFileRequestHandler"  );

  Object ocore = request.getAttribute("org.apache.solr.SolrCore");
  SolrCore core = ocore instanceof SolrCore? (SolrCore) ocore : SolrCore.getSolrCore();
  String fname = request.getParameter("file");
  String optional = request.getParameter("optional");
  String gettableFiles = core.getSolrConfig().get("admin/gettableFiles","");
  StringTokenizer st = new StringTokenizer(gettableFiles);
  InputStream is;
  boolean isValid = false;
  boolean isOptional = false;
  if (fname != null) {
    // Validate fname
    while(st.hasMoreTokens()) {
      if (st.nextToken().compareTo(fname) == 0) isValid = true;
    }
  }
  if (optional!=null && optional.equalsIgnoreCase("y")) {
    isOptional=true;
  }
  if (isValid) {
    try {
    is= core.getSolrConfig().openResource(fname);
    Reader input = new InputStreamReader(is);
    char[] buf = new char[4096];
    while (true) {
      int len = input.read(buf);
      if (len<=0) break;
      out.write(buf,0,len);
    }
    }
    catch (RuntimeException re) {
      if (!isOptional) {
        throw re;
      }
    }
  } else {
    out.println("<ERROR>");
    out.println("Permission denied for file "+ fname);
    out.println("</ERROR>");
  }
%>

from the code, it will check the configuration of gettablefiles in solrconfig.config  by default, there is no such configration in the solrconfig.xml (which comes with the new verson), so it means those files are not getable by default. Why?

the newer version expose this file get functionality via another url called /admin/file?
  which is configed in solrconfigxml

 image

we get the answer , so fix is easy. just put one more entry in the admin section of solrconfig.xml, rememer to restart solr also.

image

Now we can use the solrsharp to upload some documents to solr.

image

For more information.

Monday, September 20, 2010

Important ASP.NET Security Vulnerability for ALL versions. and all app like sharepoint, dnn, reporting service. all aspx pages

MSFT just released one one Microsoft Security Advisory about a security vulnerability in ASP.NET.  This vulnerability exists in all versions of ASP.NET.

what’s can be used by attacker.

  • request and download files within an ASP.NET Application like the web.config file (which often contains sensitive data).
  • even you DLL assembly in Bin directory. ( then they can use .net reflector to see all the source code. )

How to fix it.

  • When error happened ( Exception get thrown 500 or File not exists, status 404 ), DO not return Dedicated Error code. (like 404 or 500, )
    • FOR web.config, enable customers, mode to on or remoteonly. Remove the sub-error code mapping
      • <system.web>
           <customErrors defaultRedirect="GenericError.htm"
                         mode="RemoteOnly">
             <error statusCode="404"
                     redirect="404.htm"/>
              <error statusCode="403"
                     redirect="403.htm"/>

           </customErrors>
        </system.web>

      FOR Customized Handler or Http Module, Make sure status code is always 200 even there are some exception happened
        if(userNoAccess==true)
        {
        response.StatusCode = 403;
        }
         
        try
        {
        }
        catch(Exception ex)
        {
        Log.logError(ex);
        response.StatusCode = 500;
        }

        Change it to
        if(userNoAccess==true)
        {
        response.StatusCode = 200;
        }
         
        try
        {
        }
        catch(Exception ex)
        {
        Log.logError(ex);
        response.StatusCode = 200;
        }

When will MSFT release the patch.
  

  • it looks like the team is still on the investigation phase to get more details, I don't think it will take a long time.

Thursday, September 16, 2010

Clickonce deployment exception, System.ArgumentException - Value does not fall within the expected range. - Source: System.Deployment

Clickonce is one new features in .net framework 2.0 which make app deployment easy. Just right click you project, select the files and version. the framework will take care the auto updating and initial one-demand setup.

at one day, I get some problem like this.
  t4

or some user might get different error like this

RROR SUMMARY
    Below is a summary of the errors, details of these errors are listed later in the log.
    * Activation of http://appserver/app/Client/xxxx.application resulted in exception. Following failure messages were detected:
        + Value does not fall within the expected range.

COMPONENT STORE TRANSACTION FAILURE SUMMARY
    No transaction error was detected.

WARNINGS
    There were no warnings during this operation.

OPERATION PROGRESS STATUS
    * [9/16/2010 8:51:30 AM] : Activation of http://appserver/app/Client/xxxx.application has started.
    * [9/16/2010 8:51:30 AM] : Processing of deployment manifest has successfully completed.

ERROR DETAILS
    Following errors were detected during this operation.
    * [9/16/2010 8:51:30 AM] System.ArgumentException
        - Value does not fall within the expected range.
        - Source: System.Deployment
        - Stack trace:
            at System.Deployment.Application.NativeMethods.CorLaunchApplication(UInt32 hostType, String applicationFullName, Int32 manifestPathsCount, String[] manifestPaths, Int32 activationDataCount, String[] activationData, PROCESS_INFORMATION processInformation)
            at System.Deployment.Application.ComponentStore.ActivateApplication(DefinitionAppId appId, String activationParameter, Boolean useActivationParameter)
            at System.Deployment.Application.SubscriptionStore.ActivateApplication(DefinitionAppId appId, String activationParameter, Boolean useActivationParameter)
            at System.Deployment.Application.ApplicationActivator.Activate(DefinitionAppId appId, AssemblyManifest appManifest, String activationParameter, Boolean useActivationParameter)
            at System.Deployment.Application.ApplicationActivator.PerformDeploymentActivation(Uri activationUri, Boolean isShortcut, String textualSubId, String deploymentProviderUrlFromExtension, BrowserSettings browserSettings, String& errorPageUrl)
            at System.Deployment.Application.ApplicationActivator.ActivateDeploymentWorker(Object state)

COMPONENT STORE TRANSACTION DETAILS
    * Transaction at [9/16/2010 8:51:30 AM]
        + System.Deployment.Internal.Isolation.StoreOperationSetDeploymentMetadata
            - Status: Set
            - HRESULT: 0x0
        + System.Deployment.Internal.Isolation.StoreTransactionOperationType (27)
            - HRESULT: 0x0

When I run the cordbg and attach to the dfsvc.exe . the stack trace tells somthing different.  no signature
image

all these errors are caused by one fact that the filed downloaded by click once are “Broken”, either file is deleted or the signature is incorrect.( for example, you download the file from http://servera/, then next time same file is downloaded from the loadbalancer. http://loadbalancer/ or http://virtualIP/

Fix is easy. Go to the default once click app file location.
windows 7 will be C:\Users\[userid]\AppData\Local\Apps\2.0
xp or 2003 C:\Documents and Settings\[userid]\Local Settings\Apps\2.0

delete all those folders in that path.

try to install and run the app again, the issue will disappear.  It looks like the pre-determined state of click once is broken.

How to Create a Mysql NDB cluster on a PC, windows 7 tutorial

Just like How to create a Cassandra cluster on a single PC / windows Tutorial, I will list the prerequisites and point of config , then will go through a simple cluster configuration step by step.  I will create a two storage Node + 1 Management Node + one MySQl (API Node)

before you try to run/create a cluster, please check the following requirements.

  • Two TCP Ports,make sure no apps are using these two ports. also check the firewall policy. I will put P1 and P2 as reference later on. 
    • one is for Mysql API, I will put 5000 here. ( Just like the port 3306 for the standard MySQL.  the port will used for the connection library.)
    • one is the Management Port, by default is 1186.
    • No other ports are needed to be specified explicitly. ( every data node just handshake with management node first to establish the cluster membership. like the gossip-based clustering protocol.)
  • TWO Data Folders, DataFolder_1 and DataFolder_2
    • one is for Data Node. (ALL Data Nodes  in the same host will share the same folder by default. )
    • another one is for API Node ( just like the data folder in the standalone mysql, user management , information schema, those system tables.)
  • TWO Config files. Config_1 and Congfig_2
    • one is for Management Node.
    • another one is for API Node
    • NO config file for data node directly. why? when data node join the cluster, it will first talk to management node. the manage node will tell data node what it’s configuration will be.)
  • One Config folder, Conf_1
    • a central place to store Confg_1 and Config_2 , also the system generated config.

then, Let’s start the installation process. basically just copy and past files. :)

1. Download the mysql cluster bits from mysql.com. http://www.mysql.com/downloads/cluster/. My base folder is C:\mysql\mysqlcluster

unzip it , and rename the uncompressed folder . I will put it to C:\mysql\mysqlcluster\mysqlc. so the folder structure will looks like these.

image

Add the Bin folder to the system path. then you dont have to key in the full path to execute those commands.

2. Create DataFolder_1 and Data_folder2. and One Config folder.

Data Node will be C:\mysql\mysqlcluster\my_cluster\ndb_data
   Mysql data node will be C:\mysql\mysqlcluster\my_cluster\mysqld_data
  onc folder, C:\mysql\mysqlcluster\my_cluster\conf

copy the initial Mysql and Ndbinfo folder from the downloaded data folder, here will be C:\mysql\mysqlcluster\mysqlc\data to Mysql data folder. #2 folder above.

3. Create two config file. 
   Config_1 for management node. I will put it into C:\mysql\mysqlcluster\my_cluster\conf\management.ini
 

[ndb_mgmd]
hostname=localhost
datadir=C:\mysql\mysqlcluster\my_cluster\ndb_data
id=1
[ndbd default]
noofreplicas=2
datadir=C:\mysql\mysqlcluster\my_cluster\ndb_data
[ndbd]
hostname=localhost
id=3
[ndbd]
hostname=localhost
id=4
[mysqld]
Id=50

then the config_2 for mysql API Node.

I will put it into C:\mysql\mysqlcluster\my_cluster\conf\mysqld.conf

[mysqld]
ndbcluster
datadir=C:\mysql\mysqlcluster\my_cluster\mysqld_data
basedir=C:\mysql\mysqlcluster\mysqlc
port=5000

4. time to kick off the Cluster.

A. Start the management Node first. ( run the initial loading. , always to be the 1st Node. waiting for all nodes to connect and join the cluster.)

ndb_mgmd -f c:\mysql\mysqlcluster\my_cluster\conf\management.ini --initial --config-dir=c:\mysql\mysqlcluster\my_cluster\conf

  
after that, when you check the conf foder. one system config file will be generated.
image

Here by default, ndb_mgmt process will listen on the 1186 port and waiting for other node to join.
image

When you run the “ndb_mgm -e show” command, it will you the cluster status. at this moment, Only management Node is Ready.  other nodes are disconnected.


image

B. Start Storage Node 3. ( all you need is to specify the Management Node endpoint, If you get fail to spawn process error, put the foreground option here. )
       run “ndbd -c localhost:1186 --foreground=true”

   image

run status again, “ndb_mgm –e show”, Node 3 is up now.
image

Let’s start Node 4. run the same command. “ndbd -c localhost:1186 --foreground=true”. here is the trick. we dont specify the node id. we will talk to the mgmt node. it will assign one for you.
run status again, “ndb_mgm –e show”, Bote Nodes are ready

image

C. Start the API Node. ( which is equivalent to the mysqld in standalone mysql.)

mysqld --defaults-file="c:\mysql\mysqlcluster\my_cluster\conf\mysqld.conf"

after that, we are done. all nodes are up

image

AS a Say, all talks to Center management port , you can tell from the tcpviewer.

image

Now cluster is ready, let’s connect to the API node. and create some data.

mysql -h 127.0.0.1 –P5000 -u root
mysql> create database clusterdb;use clusterdb;
mysql> create table simples (id int not null primary key) engine=ndb;
mysql> insert into simples values (1),(2),(3),(4);
mysql> select * from simples;

here by default , the datamemory and indexmemory will determined the db capacity. because all data are stored in memory by default.

you can try push more data to the db. and query the system table.

image

since 5.1.16, we can offload the non-indexed column from memory to disk. here is the basic syntax.

CREATE LOGFILE GROUP lg_1
    ADD UNDOFILE 'undo_1.log'
    INITIAL_SIZE 16M
    UNDO_BUFFER_SIZE 2M
    ENGINE NDBCLUSTER;

    CREATE TABLESPACE ts_1
    ADD DATAFILE 'data_1.dat'
    USE LOGFILE GROUP lg_1
    INITIAL_SIZE 32M
    ENGINE NDBCLUSTER;

    create table nonmemory2(junk char(255) default null) tablespace ts_1 storage disk engine=ndb;

also, please note, even data is stored in disk, it still need a small memory footprint for each record. around 8 bytes as a point to disk.

More tutorials.
How to create a Cassandra cluster on a single PC
how to setup multi ip address on one network card
Lucene, Indexing and searching

Wednesday, September 15, 2010

programmatically retrieve SQL Server stored procedure source code, query wheter the code contains some keyword.

Here is a quick script to receptive the SQL Server sp source code, Or query whether it contains some keyword or Not. the keyword could be a table name, snippet. etc.

create table #test1( text varchar(2000))
create table #test2( text varchar(2000))
insert into #test1 select SCHEMA_NAME(schema_id) + '.'+name from sys.procedures

DECLARE db_cursor CURSOR FOR 
select text from #test1
declare @name varchar(3000)
OPEN db_cursor  
FETCH NEXT FROM db_cursor INTO @name  
WHILE @@FETCH_STATUS = 0  
BEGIN  
       insert into #test2   exec sp_HelpText @name
       if exists(select * from #test2 where  text like '%yourkeyword%')
        print 'found,procedure is '  + @name
        FETCH NEXT FROM db_cursor INTO @name  
END  

CLOSE db_cursor  
DEALLOCATE db_cursor
drop table #test1
drop table #test2

If you query all adventoreworks sp for AS keyword. you will get

(11 row(s) affected)
found,procedure is dbo.uspPrintError

(43 row(s) affected)
found,procedure is dbo.uspLogError

(31 row(s) affected)
found,procedure is dbo.uspGetBillOfMaterials

(30 row(s) affected)
found,procedure is dbo.uspGetEmployeeManagers

(30 row(s) affected)
found,procedure is dbo.uspGetManagerEmployees

(31 row(s) affected)
found,procedure is dbo.uspGetWhereUsedProductID

(35 row(s) affected)
found,procedure is HumanResources.uspUpdateEmployeeHireInfo

(24 row(s) affected)
found,procedure is HumanResources.uspUpdateEmployeeLogin

(22 row(s) affected)
found,procedure is HumanResources.uspUpdateEmployeePersonalInfo

Tuesday, September 14, 2010

Fix WCF AddressFilter mismatch error, customized IServiceBehavior , WCF service behind Load balancer or Firewall

If you WCF service is hosed behind Load balancer or firewall , or been forwarded by any kind of device. you may end with an error like

The message with To 'http://127.0.0.1:8888/bHost1/WCFService1.svc' cannot be processed at the receiver, due to an AddressFilter mismatch at the EndpointDispatcher.  Check that the sender and receiver's EndpointAddresses agree.


if you have the source code, you may have to put one attribute to each of those services.

[ServiceBehavior(AddressFilterMode=AddressFilterMode.Any)]

that’s fine if you own the source code. or just handful of service. If you have hundreds of service, that will be terrible change. How to do that in that case.

answse is pretty simple, write a behavior and apply it to the configuration. All you have to do is writing a behavior which will change the default addressfilter, and Apply the behavior by changing the web.config or app.config

using System;
using System.Collections.Generic;
using System.Text;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Configuration;

namespace Androidyou.TestLib
{
    public class OverrideWCFAddressFilterServiceBehavior : IServiceBehavior
    {
        public void AddBindingParameters(ServiceDescription serviceDescription, System.ServiceModel.ServiceHostBase serviceHostBase, System.Collections.ObjectModel.Collection<ServiceEndpoint> endpoints, System.ServiceModel.Channels.BindingParameterCollection bindingParameters)
        {

        }

        public void ApplyDispatchBehavior(ServiceDescription serviceDescription, System.ServiceModel.ServiceHostBase serviceHostBase)
        {
            for (int i = 0; i < serviceHostBase.ChannelDispatchers.Count; i++)
            {
                ChannelDispatcher channelDispatcher = serviceHostBase.ChannelDispatchers[i] as ChannelDispatcher;

                foreach (EndpointDispatcher dispatcher2 in channelDispatcher.Endpoints)
                {
                    dispatcher2.AddressFilter = new MatchAllMessageFilter();
                }
            }
        }

        public void Validate(ServiceDescription serviceDescription, System.ServiceModel.ServiceHostBase serviceHostBase)
        {

        }
    }
    public class OverrideAddressFilterModeElement : BehaviorExtensionElement
    {

        public override Type BehaviorType
        {
            get { return typeof(OverrideWCFAddressFilterServiceBehavior); }
        }

        protected override object CreateBehavior()
        {
            return new OverrideWCFAddressFilterServiceBehavior();
        }
    }
}


Config change.

 

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <system.serviceModel>
        <behaviors>
            <serviceBehaviors>
                <behavior name="newbehavior">
                    <matchalladdressfilter/>
                </behavior>
            </serviceBehaviors>
        </behaviors>
        <extensions>
            <behaviorExtensions>
               <add name="matchalladdressfilter" type="Androidyou.TestLib.OverrideAddressFilterModeElement, Androidyou.TestLib, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />
            </behaviorExtensions>
        </extensions>
      
        <services>
            <service name="ConsoleApplication4.SayService" behaviorConfiguration="newbehavior">
              <host>
                <baseAddresses>
                  <add baseAddress="http://localhost:9999/svc"/>
                
                </baseAddresses>
              </host>
                <endpoint address="ws" binding="wsHttpBinding" bindingConfiguration="wsbinding"
                    contract="ConsoleApplication4.SayService" />
            </service>
        </services>
    </system.serviceModel>
</configuration>

Or you may apply the behavior in Code.


host.Description.Behaviors.Add(new Androidyou.TestLib.WCFSecurityLib.OverrideWCFAddressFilterServiceBehavior());

if you want to change addressfiltermode programmatically, you may put a switch into the above behaviors, like a static variable, then change the switch programmatically which will cause the behavior to refresh the filtermode.

Friday, September 10, 2010

Biztalk 64bit Host CPU 100% , Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread

I’ve been involved to nail down one CPU 100% issue. per my understanding, the ETW trace API failed and caused the System to run the refresh in an infinite loop.

what’s the symbol of the event?

there are 9 HOSTs in one Biztalk server. only 1 Host kept hitting 100% cpu. even there was no business activity. ( no message came in, no orchestration get invoked. )

what did I do to locate the problem. 
   
  Run a memory Dump, then open the dump in Windbg, see which thread is CPU hungry.

run !runaway
 
91:1235      0 days 3:35:28.796
92:9740      0 days 3:34:24.140

threads 91 and 92 are the top threads which consume a lot CPU time.

then Run a clrstack to see what are those two threads doing.

!91s //switch to thread 91
!clrstack –a //view the stacktrace, and show the instance of the variable
Both 91 and 92 get the same result.

00000000143fc810 00000642783437fa System.Diagnostics.StackTrace.ToString(TraceFormat)
00000000143fc8f0 00000642808b34e6 System.Exception.get_StackTrace()
00000000143fc930 000006427f535468 Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread()
00000000143fe850 00000642808b3ac7 Microsoft.BizTalk.MsgBoxPerfCounters.MgmtDbAccessEntity.UpdateDeltaMACacheRefreshInterval()
00000000143fe8e0 00000642808b32b7 Microsoft.BizTalk.MsgBoxPerfCounters.MgmtDbAccessEntity.RunCacheUpdates(Int32)
00000000143fe970 00000642782f173b Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread()
00000000143fea10 000006427838959d System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
00000000143fea60 000006427f602672 System.Threading.ThreadHelper.ThreadStart()

Let’s see the stacktrace from bottom up, the closed Biz code is the Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager.RunCacheThread
the update by default should be invoked every 60 seconds. why it keep invoking. ???

since memory dump is a static analysis and snapshot of the memory. then a run a cordbg,  what surprised me is that the process keep throwing exceptions.
like this

first chance exception generated: (0xe00a18b8) <System.Threading.ThreadAbortExcetion>
first chance exception generated: (0x1600a0e70) <System.Threading.ThreadAbortExcption>
first chance exception generated: (0xe00a2ff0) <System.Threading.ThreadAbortExcetion>
first chance exception generated: (0x1600a21b0) <System.Threading.ThreadAbortExcption>
first chance exception generated: (0xe00a4580) <System.Threading.ThreadAbortExcetion>
first chance exception generated: (0x1600a38d8) <System.Threading.ThreadAbortExcption>
first chance exception generated: (0xe00a58d8) <System.Threading.ThreadAbortExcetion>
first chance exception generated: (0x1600a4e60) <System.Threading.ThreadAbortExcption>
first chance exception generated: (0xe00a7010) <System.Threading.ThreadAbortExcetion>
first chance exception generated: (0x1600a61a0) <System.Threading.ThreadAbortExcption>
first chance exception generated: (0xe00a85a0) <System.Threading.ThreadAbortExcetion>

then I turned on the “unhanled exceptino on” by run “ca e” get the exception details

first chance exception generated: (0x120128fd0) <System.Threading.ThreadAbortExc
ption>
_className=<null>
_exceptionMethod=<null>
_exceptionMethodString=<null>
_message=(0x120129058) "Thread was being aborted."
_data=<null>
_innerException=<null>
_helpURL=<null>
_stackTrace=(0x1201290a8) <System.SByte[]>
_stackTraceString=<null>
_remoteStackTraceString=<null>
_remoteStackIndex=0
_dynamicMethods=<null>
_HResult=-2146233040
_source=<null>
_xptrs=0
_xcode=-532459699
xception is called:FIRST_CHANCE
ative disassembly not available.
cordbg)

is it the same stacktrace, yes it is. run W

)* Microsoft.BizTalk.MsgBoxPerfCounters.MgmtDbAccessEntity::UpdateDeltaMACacheR
freshInterval
+0171[native] +0035[IL] in <Unknown File Name>:<Unknown Line Numb
r>
)  Microsoft.BizTalk.MsgBoxPerfCounters.MgmtDbAccessEntity::RunCacheUpdates +01
1[native] +0039[IL] in <Unknown File Name>:<Unknown Line Number>
)  Microsoft.BizTalk.MsgBoxPerfCounters.CounterManager::RunCacheThread +0279[na
ive] +0093[IL] in <Unknown File Name>:<Unknown Line Number>
)  System.Threading.ExecutionContext::Run +0155[native] +0095[IL] in <Unknown F
le Name>:<Unknown Line Number>
)  System.Threading.ThreadHelper::ThreadStart +0077[native] +0025[IL] in <Unkno
n File Name>:<Unknown Line Number>
)  [Internal Frame, 'AD switch':(AD ''. #) -->(AD '__XDomain_3.0.1.0_0'. #7)]

then I use the ILDASM , and Open the assembly, go the the 0x 23 Lines. 

IL_001c:  callvirt   instance void [Microsoft.BizTalk.Tracing]Microsoft.BizTalk.Tracing.Trace/HackTraceProvider::TraceMessage(uint32,
                                                                                                                               string,
                                                                                                                               object[])
IL_0021:  ldnull
IL_0022:  stloc.2d


it is a ETW trace API.

Trace.Tracer.TraceMessage(4, "MgmtDbAccessEntity: Entering RunCacheUpdates fn for host " + this.hostName, new object[0])


what a bad design for the code that Tracing API (non-functional) broke the functional application, and even didn't catch the unhandled exception

More troubleshooting entries.

Free WCF / ASMX Test Trace tool. SOAPbox by vordel

in Free WCF / ASMX Test Trace tool. SoapUI and TCPTrace, I explained the features of SoapUI. besides, SoapUI, SOAPBox is also one great tool to be used as a wcf/asmx test tool. SoapBox is offered for free download by the xml gateway vendor vordel.

image

Download and install from the link , http://www.vordel.com/products/soapbox/GetSOAPbox.html

Given a very basic Service.

image

when run the host application, you should be able to access the WSDL.

image

When you start the SOAPbox , the screen looks as bellow.

image

Click File->Import WSDL. you can browse the WSDL file locally or enter the url.

then select the operations available in the WSDL to test

image

then Click the blue arrow button to run.

image

hint: right click the content panel, click format to make the xml more user-friendly to view.

you can click the service and change the Port Url.

image

if the service requires a basic authentication, you can input the token here.

image

also you may chose Kerberos authentication.

for rest service or http testing, you may change the http verb.

image

for more security features, like Encryption, signature.

image

 

Conclusion, SOAPBox has more security features than SoapUI. while SOAPUI has more support on different protocols. like jms, jdbc, etc.

more FREE tools for application developers and system administrators.

Wednesday, September 8, 2010

Free MAC HFS file system viewer on windows

When you’ve installed windows 7 on a Macbook, and always switch between two Mac and windows. you may have noticed that you can’t view the MAC file system on windows 7.

when you open the disk manager utility, the partition of mac system is unknown.

image

what’s the gap ?  Windows is based on NTFS/Fat, while MAC uses different file system called HFS. By default, windows doesn’t recognize and understand the file system format, Microsoft might meant to do that. there is a PC , why Mac:)

with the help of a free utility called HFSexplorer. you can read the MAC file system in windows.

simply download the zip and extract the bits. click the HFSexplorer.exe please make sure java runtime is installed on the pc, the utility is based on java technology.

image

Click file->load file from device.

image

click load , then you will be able to navigation the Mac system.

image 

here you have to select the file or folder, click the EXTRACT button to transform the file to windows format.

 

more FREE tools for application developers and system administrators.

Hello Lucene, Indexing and searching

I am reading the book Lucene in Action, Second Edition: Covers Apache Lucene 3.0. in the chapter one, there is one basic java program which do the 101 indexing and searching. 

  Here are some basic tutorial to do that.
  1. there is only one core jar file necessary for the engine to run, you can download it from http://www.apache.org/dyn/closer.cgi/lucene/java/

2. open the eclipse , create one java project and reference the core jar file.

3. Create a text file , and put some contents. then save it as test.txt

4. write some java code to do the indexing and searching.
 

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.Field.Index;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.document.Fieldable;
import org.apache.lucene.index.*;
import org.apache.lucene.index.IndexWriter.MaxFieldLength;
import org.apache.lucene.queryParser.QueryParser;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.store.LockObtainFailedException;
import org.apache.lucene.util.Version;

public class Program {

    public static void main(String[] args) {
        try {
            IndexFile("/Users/androidyou/Documents/lucence/data/test.txt",
                    "/Users/androidyou/Documents/lucence/index");

            Search("/Users/androidyou/Documents/lucence/index","nonexistedkeyworld");
            Search("/Users/androidyou/Documents/lucence/index","apache");

        } catch (Exception e) {
            // TODO Auto-generated catch block
        }
        System.out.println("done");
    }

    private static void Search(String indexpath, String keyword) throws Exception, IOException {
        IndexSearcher searcher=new IndexSearcher(FSDirectory.open(new File(indexpath)));
        System.out.println("Search  keyword " + keyword);
        Query query=new QueryParser(Version.LUCENE_30, "content", new StandardAnalyzer(Version.LUCENE_30)).parse(keyword);

        TopDocs docs= searcher.search(query, 10);
        System.out.println("hits " + docs.totalHits);
        for(ScoreDoc doc: docs.scoreDocs)
        {
            System.out.println("doc id" + doc.doc + "doc filename" + searcher.doc(doc.doc).get("filename")) ;
        }

    }

    private static void IndexFile(String datafolder, String indexfolder) throws CorruptIndexException, LockObtainFailedException, IOException {
        Analyzer a=new StandardAnalyzer(Version.LUCENE_30);
        Directory d=FSDirectory.open(new File(indexfolder));
        MaxFieldLength mfl=new MaxFieldLength(4000);
        IndexWriter iw=new IndexWriter(d, a, mfl);

        Document doc=new Document();
        Fieldable contentfield=new Field("content", new FileReader(datafolder));
        doc.add(contentfield);
        Fieldable namefield=new Field("filename",datafolder, Store.YES, Index.NOT_ANALYZED);
        doc.add(namefield);

        iw.addDocument(doc);
        iw.commit();

    }
}

And here, if you run the program three times, there will be three “Documents” in the index repository.
  here, I will get

Search keyword nonexistedkeyworld
hits 0
Search keyword apache
hits 3
doc id 0 doc filename/Users/androidyou/Documents/lucence/data/test.txt
doc id 1 doc filename/Users/androidyou/Documents/lucence/data/test.txt
doc id 2 doc filename/Users/androidyou/Documents/lucence/data/test.txt
done

also you can download the lucene toolkit luke. and Open the index directory.

a1

from the snapshoot above, you can see there are 3 documents inside the Index. for each document, it has two fields. totally 58+1=59 terms

for the content field. by default . method Field(name , filereader) only index the field, not store it.
  when you click Documents tab, you can browse the document individually.  also you can verify that only filename is stored in the index. for the content filed, just terms. (indexed content.)

Screen shot 2010-09-08 at 11.26.41 AM

Friday, September 3, 2010

Update CSV File to Solr

By Default, Solr has several RequestHandlers that been mapped to different URLs.

you can check these settings in the solr.xml

image
so you can post document (xml format) to /update
or CSV document to /update/csv

Here is one quick example.

Let's say you have a simple CSV file named EE.csv

id;title
1;AndroidYou
2;David

post via CSV
run

curl "http://localhost:8080/solr/update/csv?commit=true&separator=%3b" --data-binary @ee.csv -H "Content-Type:text/plain"

then you may query

http://localhost:8080/solr/select/?q=id:2 OR id:1 to see these documents



 
Locations of visitors to this page