06/24 Emergency Server Maitance was not able to fix the problem

[Sizustar]

06/24's maitance effect won't be felt by most people, and Sega updated max connection again to try to fix this issue.

Source
https://twitter.com/sega_pso2/status/216902170824544256

[Scyris]

The problem is the fact that too much stuff is server side that shouldn't/doesn't need to be. For example, does every npc need to have all their text be server side? Nope. Do all missions and descriptions need to be server side for the text? nope, Just whether you can access that mission or not.

Its seems to me almost everything other than models, textures and music is server side, that right there is where the lag is coming from, its not network lag they are having, its the cpu getting throttled due to too much stuff going at once. If it was the connection I'd DC but I have never, ever been dced once. Seen this lag in other mmo's its a harware issue, there are really 3 soltuions: limit amount of people on (or implement a queue system of some kind), or, remove some of the stuff thats server side that does not need to be (npc text and the like, mission descriptions etc), and lastly but least likely to get done: upgrade the server hardware.

Most companys won't go for #3 on a newly released title, because of the mass amount of people interested, it usually tapers off after about a week or so.

For now I've stoped playing other than to login daily for the FUN points, the game is unplayable with 15-20 second lag spikes every minute or 2.

[renrinken]

Well its beta after all, I am no server engineer so I am not qualified to determine exact causes of lag. But they seem to be doing their job to fix it. If cause is indeed as Scyris says, most likely they will alter data by release.

[Krisan]

I'm just happy they actually TRIED to place things on the server end for once. With previous games they left far too much up to the client.. Though admittedly, it's quite silly how much text of all things is left up to the server.

[VonHalford]

I don't see the logic in how people think Text causes lag. I know you want an english patch badly but you can't just make up science. Text Files are so incredibly small.

[Ryuujin13]

imo they should just leave all of the text on the client side and then keep all of the stats server side. I mean besides translating the text who is really going to want to mod the text to say something silly? However as for lag they do need to just add more blocks. There are more people than the blocks can handle at the moment.

[Thermalwolf]

I don't see the logic in how people think Text causes lag. I know you want an english patch badly but you can't just make up science. Text Files are so incredibly small.

^^^this...

Seriously leaving all the text files server sided does not create lag. This is obviously just not enough server power to handle all the client requests at once so it lags. I'm sure Sega will have an answer soon enough.

[renrinken]

Ultimately the Engineers will determine exact source of lag. Random players without any formal education certainly are least qualified to figure out source. If it was so straight forward then it be fixed already, especially sense each one of these emergency maintenance just cost SEGA more and more money.

[Thermalwolf]

Ultimately the Engineers will determine exact source of lag. Random players without any formal education certainly are least qualified to figure out source. If it was so straight forward then it be fixed already, especially sense each one of these emergency maintenance just cost SEGA more and more money.

Sometimes the problem is straight forward but it's not as simple as attacking it. There's a lot that goes into keeping a server up an running. Lots of protocols and MySQL, DNS Servers, Name Servers, and other things that the connections have to route through before the server is open to the internet. Perhaps it's in the DNS Servers. One change could mean they have to completely rework and program there server structure.

I'm sure they have a damn good idea of whats creating these issues, it's fixing it that's difficult.

[Vylera]

Let's face it.

If they IP blocked the game from the start, they wouldn't have needed 3 maintanances already.

[Zyrusticae]

It's more than likely that the server hardware itself is inadequate to handle the sheer number of calculations being thrown at it at once.

This is not an easy problem to solve, mind you. For example, an often bandied-about solution is to increase the use of multithreading, but that requires that the code be programmed from the outset for parallelism - rewriting code to go from two, to three, to four threads is not at all a trivial task, and in some cases may simply be impossible (for example, in cases where data from one thread is dependent on receiving data from another, possibly causing stalling).

I have no idea how they handle this. It could be that each block is being handled by a single server shard, which would be a terrible idea and could be the cause of the heavy latency. But then, with 500 blocks out there, I can't say I'd blame them for taking that kind of route...

This is just speculation, mind you. I don't know the internal workings of Sega's server structure, I only know from experience with other, likely much smaller, server setups.

[Chik'Tikka]

It's more than likely that the server hardware itself is inadequate to handle the sheer number of calculations being thrown at it at once.

This is not an easy problem to solve, mind you. For example, an often bandied-about solution is to increase the use of multithreading, but that requires that the code be programmed from the outset for parallelism - rewriting code to go from two, to three, to four threads is not at all a trivial task, and in some cases may simply be impossible (for example, in cases where data from one thread is dependent on receiving data from another, possibly causing stalling).

I have no idea how they handle this. It could be that each block is being handled by a single server shard, which would be a terrible idea and could be the cause of the heavy latency. But then, with 500 blocks out there, I can't say I'd blame them for taking that kind of route...

This is just speculation, mind you. I don't know the internal workings of Sega's server structure, I only know from experience with other, likely much smaller, server setups.

+^_^+ I'd love to get a look at Sega's hardware+^_^+ I've been trying to get a peek at AT&T's local NOC for awhile, it's tucked behind this bar uptown in a squat gray building, but the things a vault (i heard rumors of water cooled servers in there, with all the cabling hanging from the ceiling and the water running underneath)+^_^+

[moeri]

I don't see the logic in how people think Text causes lag. I know you want an english patch badly but you can't just make up science. Text Files are so incredibly small.

hmm, though leaving all of that server side would atleast cause unnecessary stress on the servers

for example...

a simple quest description could easily be 1KB

lets say a million requests are made at the same time, this would be 1GB of data right off the start, that then needs to be transfered to the players

now, to tell the truth this is no big deal, the real problem is that with all the text on the server side, requests for the data need to be handled, which will put a lot more strain then the simple text files themselves, requesting the particular text, looking it up, handling the request, and such could easily be a big bottleneck, but they seem to have this handled...

putting the text client side would probably be a very good idea for them, but whether the actual source of lag is there, is shaky ground to stand on to say the least... I see no reson to keep the text server side, but meh, that is their choice

again, a bunch of text requests are probably nothing for sega's servers/ optimizations

[Polantaris]

Let's face it.

If they IP blocked the game from the start, they wouldn't have needed 3 maintanances already.

lolwhat?

Foreign connections don't cause lag. The servers handle requests the same regardless of source unless they implement something otherwise, which they have not.

[moeri]

lolwhat?

Foreign connections don't cause lag. The servers handle requests the same regardless of source unless they implement something otherwise, which they have not.

I" am pretty sure they mean that the servers would not have to deal with as many requests(as many people), therefore not causing a pile up of requests, causing lag

[Polantaris]

I" am pretty sure they mean that the servers would not have to deal with as many requests(as many people), therefore not causing a pile up of requests, causing lag

Except there's not that many foreigners playing when you compare to the amount of local players.

No matter how you want to look at it, foreigners are not the cause of the lag. Every single test has had serious lag issues (Minus possibly A1, I wasn't in that one), and in both A2 and CBT they admitted to some sort of server hardware issues.

[moeri]

Except there's not that many foreigners playing when you compare to the amount of local players.

Do you have numbers to back that up? If foreigners were even a small percentage of the population, then this might account for an unexpected number of people playing.

say... 10% would be enough to cause estimations of number of people playing to be far enough off to cause some issues such as the lag being experienced




Every single test has had serious lag issues (Minus possibly A1, I wasn't in that one), and in both A2 and CBT they admitted to some sort of server hardware issues.

Yes, but this could be due to an unexpected number of people? They really didn't limit the number of people that could play... atleast in CBT

"some sort of server hardware issues", could easily be a cover for, "wow we didn't expect this many people"

I am not trying to say that foreigners are the issue, but just knocking the possibility out of the way without proper testing/ analysis, would be a mistake

[Sizustar]

Do you have numbers to back that up? If foreigners were even a small percentage of the population, then this might account for an unexpected number of people playing.

say... 10% would be enough to cause estimations of number of people playing to be far enough off to cause some issues such as the lag being experienced

In order to make your statement true, you also need to provide evidence.

[moeri]

In order to make your statement true, you also need to provide evidence.

I am sorry let me correct my english

let us say... 10% would be enough to cause estimations of number of people playing to be far enough off to cause some issues such as the lag being experienced, maybe 10% of the population are foriegners

I am sorry, but with that example I was only trying to show how speculation can be thrown around, but without numbers to back up your speculations, they will forever only be speculations, I was only trying to show I could throw out a counter argument, as long as neither of us have hard numbers, both of our arguments are fairly weak and have no true value

Afterward, I went on to just reinforce the fact that a "possible" cause for a problem should not be dismissed without proper evidence, though I will say that this being that cause is very unlikely, but dismissing it entirely is the mistake.

[Zyrusticae]

Well, if we assume 90% of the foreigners on Ship 2 are playing in blocks 20->2x, right now we only fill up three and a half blocks.

The locals fill up all fifty.

So yeah, pretty sure we're a tiny, insignificant fraction. Furthermore, we do NOT play during peak Japanese hours (well, most of us, anyways). It would be disingenuous to even suggest we could be high contributors to this problem.

[Scyris]

^^^this...

Seriously leaving all the text files server sided does not create lag. This is obviously just not enough server power to handle all the client requests at once so it lags. I'm sure Sega will have an answer soon enough.

Again if text wasn't server side there would be less strain on the server, it uses low bandwith but the cpu and memory has to work to make sure everyone gets sent the right text.

[renrinken]

And even then, part of the 3 blocks we us are also fill up have JP players. Any rational person understands that region locks would not fix the problem.

As I said earlier, let the Engineers figure the real problem. Watching uneducated people arguing back and forth about server architecture is just plain embarrassing.

[Scyris]

I" am pretty sure they mean that the servers would not have to deal with as many requests(as many people), therefore not causing a pile up of requests, causing lag

Um I been on during japanese player hour, and during JP time there is about 3x as many people on as there is during the time most english speaking players are in the game. So the problem would be there whether non-japan residents were ip blocked or not. During JP hour all blocks are 75% to full. English player hour usually 20 maybe 21 is full, the rest are lower than 25% usually, this is on ship 2 where most of the english speaking playerbase resides.

[moeri]

Any rational person understands that region locks would not fix the problem.
.

Why? Less people should equate to less server load xP (let us not argue that) I really doubt region locks would help as well


As I said earlier, let the Engineers figure the real problem. Watching uneducated people arguing back and forth about server architecture is just plain embarrassing.

This, but really people just want to talk about the issues, really this is probably just a way for them to blow off steam, because they are frustrated with not being able to play pso2.



Um I been on during japanese player hour, and during JP time there is about 3x as many people on as there is during the time most english speaking players are in the game. So the problem would be there whether non-japan residents were ip blocked or not. During JP hour all blocks are 75% to full. English player hour usually 20 maybe 21 is full, the rest are lower than 25% usually.

I 100% agree that it is probably not the foreigners causing the issue. The problem is we don't know the extent to which the servers are overloaded, without knowing this it is fairly possible that a small portion of the population disappearing could significantly increase performance. I am not saying that it will make a difference, but that there is a small possibility.

[Chik'Tikka]

Yes, but this could be due to an unexpected number of people? They really didn't limit the number of people that could play... atleast in CBT

"some sort of server hardware issues", could easily be a cover for, "wow we didn't expect this many people"

I am not trying to say that foreigners are the issue, but just knocking the possibility out of the way without proper testing/ analysis, would be a mistake

i thought i read somewhere that Sega stated that the system could hold 100K in CBT (5 servers) which to me means 60-80k, however the CBT lag happened with less then 30K people logged in at the same time+^_^+ so i don't think this is it+^_^+ but that's just my humble opinion+^_^+

[moeri]

i thought i read somewhere that Sega stated that the system could hold 100K in CBT (5 servers) which to me means 60-80k, however the CBT lag happened with less then 30K people logged in at the same time+^_^+ so i don't think this is it+^_^+ but that's just my humble opinion+^_^+

IF that be true then weird things are happening

[Scyris]

i thought i read somewhere that Sega stated that the system could hold 100K in CBT (5 servers) which to me means 60-80k, however the CBT lag happened with less then 30K people logged in at the same time+^_^+ so i don't think this is it+^_^+ but that's just my humble opinion+^_^+

They prob theory crafted but they probally didn't consider how much cpu/mem use having almost everything in the game server side actually is. I still think the lag is caused by the servers cpu/ram getting overloaded to the point it throttles people, the net connection is fine I think, or else i'd be getting dc'ed all the time with how bad I been lagging the last day or so. The text data as someone has said is small, but think about 10's of thousands of people accessing all diffrent npcs at the same time, which probally happens.

[TerminalMontage]

hmm, though leaving all of that server side would atleast cause unnecessary stress on the servers

for example...

a simple quest description could easily be 1KB



less, actually.

[BahnKnakyu]

The amount of incorrect info in this thread is staggering. Thanks to the few (renrinken, Zyrusticae, Sizutar, Polantaris) that are setting things straight.

From what the people who are working on the English patch have told me, the only text that seems to be server side is item names and descriptions. That's it. It definitely sounds like a hardware/code issue, and I don't think it's something they can fix overnight.

They can't release the game in this state though - the latency is unacceptable.

[Scyris]

The amount of incorrect info in this thread is staggering. Thanks to the few (renrinken, Zyrusticae, Sizutar, Polantaris) that are setting things straight.

From what the people who are working on the English patch have told me, the only text that seems to be server side is item names and descriptions. That's it. It definitely sounds like a hardware/code issue, and I don't think it's something they can fix overnight.

They can't release the game in this state though - the latency is unacceptable.

In that case I wonder just where the problem might be then. Btw I never said I was right, was just tossing some ideas out there. I do know that starting a convo with a npc seems to be effected by the lag. So maybe it sends something to the server saying "hey player #82746 is talking to npc #827216 currently". Anyway I am sure they will fix it, never seen a mmo where they didn't fix this kinda of lag within a week or so.

[moeri]

about my previous posts: I am only trying to say that instead of "foreigners are not the issue", people should be saying "it is highly unlikely that foreigners are the issue"
That is the only thing I am trying to say... oh and the same for server side text requests


less, actually.

acutally you are probably right, i just looked at some random text files on my computer, and yes those files were probably longer than a quest description... yes that was my mistake


The amount of incorrect info in this thread is staggering.

ehh there is a certain amount, and it is to be expected, though most of it is just speculation

[Thermalwolf]

I look at like this. Sega is a multi-million dollar company (Maybe even billion but I doubt it), that's been making games for 20+ years and Online games for atleast 12 years. I'm sure they know what kind of hardware they need. I think this comes down to possible bad server programming. This is the only thing to comes to mind anymore.

[KanekoSaya]

I hope they fix the lag soon it gets bad when I try to pick up items