Decode Oracle Coherence cache extend protocol, Tangosol.Net.RequestTimeoutException

on day, C# applications which use coherence cache extend get an Tangosol.Net.RequestTimeoutException. I spent a lot time to figure out what happened? what’s the potential reason of the requestimeout. whole stack trace

Tangosol.Net.RequestTimeoutException: request timed out after 30000 millis       at Tangosol.Net.Messaging.Impl.Request.RequestStatus.get_Response() at Tangosol.Net.Messaging.Impl.Request.RequestStatus.WaitForResponse(Int64 millis) at Tangosol.Net.Messaging.Impl.Channel.Request(IRequest request, Int64 millis) at Tangosol.Net.Messaging.Impl.Channel.Request(IRequest request) at Tangosol.Net.Impl.RemoteNamedCache.BinaryNamedCache.get_Item(Object key) at Tangosol.Util.ConverterCollections.ConverterDictionary.get_Item(Object key) at Tangosol.Net.Impl.RemoteNamedCache.get_Item(Object key) at Tangosol.Net.Impl.SafeNamedCache.get_Item(Object key) at CacheClient.CustomerProfile.button12_Click(Object sende

Question 1, is it a network problem? C# application will talk to proxy node (JVM on another machine), then proxy node talk to storage node to get the data back.
then I capture the network traffic between C# application and the Proxy jvm node. you can use tcpview to figure out which proxy node has the TCP connection established with the client.

the traffic looks good, request out, and response immediately get back. (so no firewall blackout, no package drop)

 

here, 10.1.111.22 is the client running C# application. It send out the request to Proxy Node (10.1.25.151)
   then after 0.024 seconds, it get the response back.(client send out ack in package 7)

all looks great on network level. Then it comes to my second question, are those data valid in the return.

in this special case, the server returns 38 bytes array here it will be

25:90:d0:92:cb:05:00:02:00:6b:03:4c:18:15:a4:37:01:00:4e:04:74:65:73:74:01:42:cf:ff:ce:91:b8:f7:b1:b2:ee:01:40:40 what the encoding mechanism, I can only figure out 1st byte 25 which is 37 means that package data length is 38-1=37, how about the rest?

then I use the .net reflector to read the Coherence.dll. and figure out the raw format.

[datalength][chennelid][typeid][versionid][objectitself]

it used packed int32 format here.
I write a Simple c# program. here to decode the channel id and tyepid/versionid

String s =”25:90:d0:92:cb:05:00:02:00:6b:03:4c:18:15:a4:37:01:00:4e:04:74:65:73:74:01:42:cf:ff:ce:91:b8:f7:b1:b2:ee:01:40:40”; byte[] data = new byte[(s.Length + 1) / 3]; for (int i = 0; i < data.Length; i++) { data[i] = (byte)   ( "0123456789ABCDEF".IndexOf(s[i * 3]) * 16 + "0123456789ABCDEF".IndexOf(s[i * 3 + 1])); } MemoryStream ms = new MemoryStream(data); DataReader dr = new DataReader(ms); string msg = string.Format("Package Length  {0} \n Channel ID {1} \n TypeID {2}\n Version id {3} ", dr.ReadPackedInt32(), dr.ReadPackedInt32(), dr.ReadPackedInt32(), dr.ReadPackedInt32()); MessageBox.Show(msg);

When run the code, you will see the decoded value.

it you try decode the output package sending to proxy, it has the same naming format.

Client send a package with the Channel ID and receive the response with channelID, if the channel state is maintained incorrectly in the client side, you will get the exception>
here the reflector code for client  decode message.

if the channel =null or is closed, even server returns back the response. it will cause the requestimeoutexception.
stupid code;(

you can also do the inspecting on the runtime level.
Define one filter.

public class MyFilterDump : IWrapperStreamFactory     {         public MyFilterDump()         {         }         public System.IO.Stream GetInputStream(System.IO.Stream stream)         {             System.Diagnostics.Debug.WriteLine("Get Response " + stream.Length);             DataReader dr = new DataReader(stream);             System.Diagnostics.Debug.WriteLine("Chennel ID " + dr.ReadPackedInt32());             stream.Seek(0, SeekOrigin.Begin);             return stream;         }         public System.IO.Stream GetOutputStream(System.IO.Stream stream)         {             System.Diagnostics.Debug.WriteLine("Send Request " + stream.Length);             return stream;             //did the same thing on input         }     }

then put it into client-coherence.xml and client-cache-control.xml

<coherence xmlns="http://schemas.tangosol.com/coherence"> <cluster-config> <filters> <filter> <filter-name>debugfilter</filter-name> <filter-class>ExtendLib.MyFilterDump, ExtendLib</filter-class> </filter> </filters> …. client-cache-control.xml <remote-cache-scheme>       <scheme-name>extend-dist</scheme-name>       <service-name>ExtendTcpCacheService</service-name>       <initiator-config>         <tcp-initiator>           <remote-addresses>             <socket-address>               <address>localhost</address>               <port>9999</port>             </socket-address>                       </remote-addresses>         </tcp-initiator>                <outgoing-message-handler>         <request-timeout>30s</request-timeout>       </outgoing-message-handler>     <use-filters>         <filter-name>debugfilter</filter-name>         </use-filters>

Have fun@