[metrics-dev] Fwd: [usage-stats] Daily Globus Usage-Stats Update

[Lee Liming]

Bill:

Here are some summary charts relevant to your points. The first shows  
the number of transfers per month, the second shows server hosts per  
month, with a breakdown by top level domain. Note that that the  
number of transfers seems to be somewhat related to the number of  
server hosts each month, though the relationship isn't exactly  
proportional.  (That makes sense: some servers are used more than  
others and the usage of each server also varies from month to month.)

The usual caveats apply.  We don't know how many UDP packets are  
being dropped between the servers and our receiver, and we don't know  
how many GridFTP deployments have usage reporting turned off.

        -- Lee
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GFTPpackethistogram.png
Type: image/png
Size: 9576 bytes
Desc: not available
URL: <http://lists.globus.org/pipermail/metrics-dev/attachments/20061012/fc9d9a2a/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GFTPhosthistogram.png
Type: image/png
Size: 14575 bytes
Desc: not available
URL: <http://lists.globus.org/pipermail/metrics-dev/attachments/20061012/fc9d9a2a/attachment-0001.png>
-------------- next part --------------

On Oct 12, 2006, at 10:06 AM, Lee Liming wrote:

> Forwarded for Bill, who is posted from an address that's not  
> subscribed to the list.
>
> Bill, we've been reporting on the metrics-dev list recently about  
> our work (Jarek's work) on completing the automation of the summary  
> report generators.  I have been producing summary reports for  
> GridFTP for two months running now using the report generators that  
> Erik Johnson wrote this summer, and there are a ton of charts  
> available.  We continue to have people who don't want to make these  
> reports public, which is frustrating to me and Ian.  They are cool!
>
> In fact, the number of GridFTP transfers (taken as a monthly total)  
> peaked a few months ago then dropped precipitously, and has been  
> ramping back up again over the last few months.  So what you are  
> seeing is a ramping back up to a level we were at a while ago.
>
> I will send you a pointer to the latest summary charts.
>
> And yes, the summary reports do take a long time to run.  Jarek has  
> found a few ways to improve them since you last had them, though,  
> so it's not quite as bad as it was.
>
>       -- Lee
>
>
> Begin forwarded message:
>
>> From: owner-metrics-dev at globus.org
>> Date: October 12, 2006 8:49:31 AM MDT
>> To: owner-metrics-dev at globus.org
>> Subject: BOUNCE metrics-dev at globus.org:    Non-member submission  
>> from ["William E. Allcock" <allcock at mcs.anl.gov>]
>>
>> From: "William E. Allcock" <allcock at mcs.anl.gov>
>> To: <metrics-dev at globus.org>
>> Subject: RE: [usage-stats] Daily Globus Usage-Stats Update
>> Date: Thu, 12 Oct 2006 09:49:22 -0500
>> Organization: Argonne National Laboratory
>> In-Reply-To: <20061012050002.90AB016503 at usage-stats.globus.org>
>>
>> Again, I haven't taken the time to do the statistics, but it  
>> appears we have
>> definitely had a step change in the daily number of transfers.  
>> Today and
>> yesterday may be flukes or may be yet another step change, but we  
>> had been
>> up in the 200k+ range regularly for a while, whereas we had been  
>> hanging
>> around 100K previously.  I wonder what happened...
>>
>> As an aside, I would suggest that a very high priority for the  
>> metrics work
>> (and actually in general) should be the summarization of the  
>> data.  I have
>> not done the statistics because the database is so huge that it is  
>> painful
>> to run queries.  As an example, the report that Erik Johnson ran  
>> on the July
>> data took on the order of 8 hours to run. One years worth of data  
>> at 100K
>> records per day is 36,500,000 records.  We have been running for  
>> over a
>> year, and our average is climbing.  I would argue that we need to do
>> something about this proactively before either a report, or a  
>> failure forces
>> our hand.
>>
>> There are lots of ways to do the summarization, but I have put  
>> some thought
>> into it, and would be willing to sketch out my thoughts for  
>> someone if that
>> is of use.
>>
>> Bill
>>
>>> -----Original Message-----
>>> From: owner-usage-stats at globus.org
>>> [mailto:owner-usage-stats at globus.org] On Behalf Of Globus Daemons
>>> Sent: Thursday, October 12, 2006 12:00 AM
>>> To: usage-stats at globus.org
>>> Subject: [usage-stats] Daily Globus Usage-Stats Update
>>>
>>> Got this from listener:
>>> 890000 packets received and successfully logged.
>>> 0 packets received that could not be parsed.
>>> 0 unrecognized packets.
>>> 0 packets were lost due to buffer overflow.
>>> Since Wed Oct 11 00:00:02 CDT 2006
>>>
>>> Breakdown by component:
>>> 94 mds_packets logged.
>>> 1582 rls_packets logged.
>>> 43494 gram_packets logged.
>>> 59741 c_ws_core_packets logged.
>>> 3477 Java WS Core v2 packets logged.
>>> 4427 Java WS Core v1 packets logged.
>>> 10812 rft_packets logged.
>>> GridFTP Transfers:
>>>   Internal to MCS: 2673
>>>   External to MCS: 763700
>>>
>>>
>>>
>>
>