Ontime lags on some events? #410


  • Defect
  • Accepted
Open
Assigned to _ForgeUser8455416
  • _ForgeUser6928255 created this issue Jan 10, 2014

    NOTE: If you supply copies of your .yml or console log files, please use pastebin, dropbox, or other service and provide link,  instead of copy/paste of the file into this report. Thanks!

    What Version of OnTime and Bukkit are you using?
    OnTime: 3.12.1
    Spigot: 1331

    What storage option are you using? (YML or MYSQL) ?
    MYSQL

    What permissions plugin are you using? (Groupmanager, PEX,bPermissions, etc.)
    Groupmanager

    Please provide a link to a copy of your plugins/OnTime/config.yml:
    http://pastebin.com/DfKAaE3d

    What steps will reproduce the problem?
    1. I don't know? It just happens
    2.
    3.

    What is the expected output? What do you see instead?
    No lag on ontime?

    Do you have an console log of what happened?
    http://aikar.co/timings.php?url=6722507 only that

    Did you turn on OnTime logging (/ontime logfile enable 1)?  If so, provide a link to that file too.
    http://pastebin.com/Xv8Ty1vC

    Please provide any additional information below.

    Timings: http://aikar.co/timings.php?url=6722507

  • _ForgeUser6928255 added the tags New Defect Jan 10, 2014
  • _ForgeUser8455416 posted a comment Jan 10, 2014

    Hi, While I investigate this can I ask you to disable the "online report function" and see what impact that has on the performance of your server?

    # Enable if Online player status should be tracked in MYSQL Table
    onlineTrackingEnable: false
    

    Also, from the error log it looks like your server may have been full and players were trying to join but failed to get a slot or something. Is this possible? And, what is the size of your server in terms of player slots, and while you were seeing the lag from OnTime what average number of players on line?

    Thanks.

  • _ForgeUser8455416 removed a tag New Jan 10, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 10, 2014
  • _ForgeUser6928255 posted a comment Jan 10, 2014

    Never full server, I had 40 players with 92 slots

  • _ForgeUser6928255 removed a tag Waiting Jan 10, 2014
  • _ForgeUser6928255 added a tag Replied Jan 10, 2014
  • _ForgeUser8455416 posted a comment Jan 10, 2014

    @danmark2312: Go

    Ok. If not a full server, would you have other reason to block certain users ? I'm looking at players philio15 and magnus2650 specifically. If not, the other possible issue could have been connection issues at the players end, and therefore not really an issue. I'm just trying to eliminate possible problems in the plugin, even though I don't think there is one.

  • _ForgeUser8455416 removed a tag Replied Jan 10, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 10, 2014
  • _ForgeUser603316 posted a comment Jan 10, 2014

    Is it me or doesn't OnTime lag at all in his report? He has 19.91 TPS average.

  • _ForgeUser603316 removed a tag Waiting Jan 10, 2014
  • _ForgeUser603316 added a tag Replied Jan 10, 2014
  • _ForgeUser8455416 posted a comment Jan 10, 2014

    @Nikecow: Go

    Cow,

    I agree. At 19.91 out of 20.00 TPS, he really should not be seeing any actual 'lag' in the true definition of the term.

    What concerns me though is the high percentages that some of the OnTime tasks are showing. I think that I should nip these in the bud before they show up truly lagging a server which has a much higher average utilization. Hence my willingness to engage in the investigation.

    I know you run performance measurements on your server, what are you seeing from OnTime theses days?

  • _ForgeUser8455416 removed a tag Replied Jan 10, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 10, 2014
  • _ForgeUser6928255 posted a comment Jan 10, 2014

    Well I was just told that my lag spikes are coming because of ontime and a few others? (based on the high percentage)

  • _ForgeUser6928255 removed a tag Waiting Jan 10, 2014
  • _ForgeUser6928255 added a tag Replied Jan 10, 2014
  • _ForgeUser8455416 posted a comment Jan 10, 2014

    @danmark2312: Go

    I would agree that if you are seeing lag spikes than OnTime looks like a likely culprit given the high percentages. What may be misleading is that the overall report sample you provided in itself is no showing an overall state of lag given the average TPS.

    So, we will keep working on this until we can fix eliminate OnTime as the cause, or fix the OnTime issue.

    Please see the request in my first response and disable the online reporting and see what that does to the OnTime percentages. Thanks.

  • _ForgeUser6928255 posted a comment Jan 10, 2014
  • _ForgeUser8455416 posted a comment Jan 10, 2014

    @danmark2312: Go

    Can you tell me of any changes you made? Can you provide updated versions of the OnTimeERROR.txt?

    Also, can you turn on OnTime logging and provide link to copy of that file too?

    /ontime logfile enable 1

    File: OnTimeLog.txt

  • _ForgeUser6928255 posted a comment Jan 11, 2014

    http://aikar.co/timings.php?url=6734269 - New timings with trackonline disabled

    error log: http://pastebin.com/B7RR5udH

    log file: http://pastebin.com/Z3FAuXsB

  • _ForgeUser8455416 posted a comment Jan 12, 2014

    @danmark2312: Go

    I'm not seeing anything in the log files that point to a processing problem with OnTime.

    The OnTime functions that are reporting high percentages are all accessing your MySQL database, so the time it is taking your database to respond to requests will have impact on OnTime's performance rating. Are you using a local database, or something provided by an external service?

    One thing I'd like you to do is look at your ontime-players table and look up the "ontime-data" entry and tell me what value you see in the "playtime" field.

  • _ForgeUser8455416 removed a tag Replied Jan 12, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 12, 2014
  • _ForgeUser6928255 posted a comment Jan 13, 2014

    @Edge209: Go

    external service playtime field: 238222548 (on my own user?)

  • _ForgeUser8455416 posted a comment Jan 13, 2014

    @danmark2312: Go

    No, not your own player. There will be a "player" entry which as a "player name" of "ontime-data".

  • _ForgeUser6928255 posted a comment Jan 13, 2014

    @Edge209: Go

    I dont get it? What is it supposed to look like?

    I'm using phpmyadmin to access my database. I see a playertime collumn?

  • _ForgeUser8455416 posted a comment Jan 13, 2014

    Ok in phpmyadmin you will see a column that is "playerName". There will be an entry (row) in the table where the "playerName" is "ontime-data".

    There is also a column that is "playtime".

    I want to know what is the value of "playtime" for the row where "playerName" is "ontime-data"

    MYSQL Search:

    SELECT * FROM `ontime-players` WHERE `playerName` = 'ontime-data' LIMIT 0 , 30

  • _ForgeUser8455416 removed a tag Replied Jan 13, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 13, 2014
  • _ForgeUser6928255 posted a comment Jan 14, 2014

    @Edge209: Go

    Oh.. I see... PLAYERNAME is ontime-data -.-' Damn I'm stupid... Anyways here it is:

    Playtime: 2

  • _ForgeUser6928255 removed a tag Waiting Jan 14, 2014
  • _ForgeUser6928255 added a tag Replied Jan 14, 2014
  • _ForgeUser8455416 posted a comment Jan 14, 2014

    @danmark2312: Go

    "2" is what I expected. One thing you can try, but I don't think it will make any real difference, but it won't hurt either. Via phpmyadmin, edit that cell and change the "2" to a "0", and reload your plugins or restart your server.

    This will cause OnTime to update the database with indexes to improve the performance. Those indexes should already be there but it won't hurt to run the updates again.

    When that is done the value in that cell will go back to "2", If you have OnTime console logging set at 'level 1' you will see messages in there about it making the updates to the database.

    Run this command if you want to see that console feedback, but do it before starting the process above.

    "/ontime console enable 1"

    Then I guess let it run for a day and do another performance test and see if there is any difference.

  • _ForgeUser8455416 removed a tag Replied Jan 14, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 14, 2014
  • _ForgeUser6928255 posted a comment Jan 14, 2014

    If I was to change the value to 0 is this the command? UPDATE ontime-players SET playtime='0' WHERE `playerName`='ontime-data';

  • _ForgeUser6928255 removed a tag Waiting Jan 14, 2014
  • _ForgeUser6928255 added a tag Replied Jan 14, 2014
  • _ForgeUser8455416 posted a comment Jan 14, 2014

    @danmark2312: Go

    Yes, that would be the right command, but in the phpmyadmin interface you should be able to just select the cell, type in the new value and hit enter. It may give you a warning message but you can ignore that.

  • _ForgeUser6928255 posted a comment Jan 14, 2014

    @Edge209: Go

    Can't click.. :/

    When I try the command I get this:

    1. 1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-players SET playtime='0' WHERE `playerName`='ontime-data'' at line 1
  • _ForgeUser8455416 posted a comment Jan 14, 2014

    @danmark2312: Go

    I must have a different phpmyadmin interface.

    I think you need to drop the quotes around playerName.

    And it looks like you may have had a double quote (") at the very end that should have been single.


    Edited Jan 14, 2014
  • _ForgeUser6928255 posted a comment Jan 14, 2014

    @Edge209: Go

    UPDATE ontime-players SET playtime='0' WHERE playerName='ontime-data';

    That should work? The double quote isn't there. It's just a "marking" quote? like 'this command'

  • _ForgeUser8455416 posted a comment Jan 14, 2014

    @danmark2312: Go

    This is what you need:

    UPDATE `ontime-players` SET `playtime` = '0' WHERE `playerName` = 'ontime-data'

    I suggest you cut and paste right from my post as there is a difference between the (') around the 2 & ontime-data and the (`) around the ontime-players, playtime, and playerName


    Edited Jan 14, 2014
  • _ForgeUser6928255 posted a comment Jan 15, 2014

    New timings: http://aikar.co/timings.php?url=6757536

    Alot better!

  • _ForgeUser8455416 posted a comment Jan 15, 2014

    @danmark2312: Go

    Was this after making the change to the online-data value?

    Can you now re-enable the online report, so we can see how that now acts?

  • _ForgeUser8455416 removed a tag Replied Jan 15, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 15, 2014
  • _ForgeUser6928255 posted a comment Jan 17, 2014
  • _ForgeUser6928255 removed a tag Waiting Jan 17, 2014
  • _ForgeUser6928255 added a tag Replied Jan 17, 2014
  • _ForgeUser6928255 posted a comment Jan 18, 2014

    A new timings file with online tracking enabled: http://aikar.co/timings.php?url=6774860

  • _ForgeUser6928255 posted a comment Jan 22, 2014

    hello?


    Edited Jan 23, 2014
  • _ForgeUser8455416 posted a comment Jan 22, 2014

    @danmark2312: Go

    Hello. I have not forgotten you. I am looking at some performance improvements for the online report, which seems to be the culprit for the high utilization. I'm looking to greatly reduce the number of MySQL queries which will provide a significant improvement.

    When you have online reports disabled are you still perceiving that there is a performance problem with the plugin?

  • _ForgeUser603316 posted a comment Jan 23, 2014

    @Edge209: Go

    Sorry for never replying to this, I don't get notified when someone replies :(

    In my 1 year of using OnTime it has never lagged, but then again we are not a big server.

    Still, I highly doubt it does even at larger numbers.

  • _ForgeUser8455416 removed a tag Replied Jan 23, 2014
  • _ForgeUser8455416 added a tag Waiting Jan 23, 2014
  • _ForgeUser6928255 posted a comment Feb 6, 2014
  • _ForgeUser6928255 removed a tag Waiting Feb 6, 2014
  • _ForgeUser6928255 added a tag Replied Feb 6, 2014
  • _ForgeUser8455416 posted a comment Mar 11, 2014

    In the latest update v3.12.2 I have made some improvements to improve handling of the online reports to reduce server lag related to MySQL activity. I have also added some general scheduled task improvements to better help with diagnosis of future issues.

    There are more changes I would like to make related to the online reports, so I will be leaving this ticket open until I make those.

  • _ForgeUser8455416 removed a tag Replied Mar 11, 2014
  • _ForgeUser8455416 added a tag Accepted Mar 11, 2014

To post a comment, please login or register a new account.