"one is to HAVE database and another is to OVERLOAD it"

Yes, one is to HAVE the test tools, another is to USE them ;)

"I can not emulate 10 visitors on my page in one second, for example"

You can, there's plenty of free, good stress test software - the most comprehensive I know is DieselTest.

"And it all depends again on the server, OS, hardware I run"

That just means one test is not enough; it does not mean that testing is useless.

While you can't test every hardware/software configuration, you can certainly get an idea of what kind of performance you can expect from this or that configuration, by looking at SIMILAR configurations tested by other people :)

Of course you need to test both an apache/ISS server on windows, apache on Linux, and with different databases etc. - but you don't have to do all this work yourself, you just have to direct people to the testing tools, and invite them to submit their results. There will be plenty of people interested in testing the performance of MiniBB before deciding to use it in a production environment.

Of course the software matters - it matters how much memory each running instance of a script consumes, and it matters how many SQL queries they require to fulfill a page request.

How much fuel you got only determines how far you can go, not how fast! ;)

How much fuel you got only determines how far you can go, not how fast!
You must know the structure of autos - and FAST, TOO :)

Anyway, it's always good to talk - so, take it, test it, inform about results, inform about your solution of optimization, we will happily appreciate it.

I will, but you'd save me a lot of work if you could provide me with your database filler script, please? :)

mindplay
I have working forums with hundred thousands posts. Today I'll create special forum for you there will be half million posts and you will be able to check it ;) I'll give you the url shortly.


thanks,

Also see stats - yes it loads 74 seconds - that's the only problem i see currently .. it happens because all stats information is shown on one page - so we will split it into several page in the next version and it will load 3 times faster then ;)


thanks,

Sergei,

20 seconds for a search page is, I'm sorry, completely useless - obviously the site is going to lag badly for all other users during those 20 seconds, and imagine what happens if two or three users search at the same time ... obviously then it's sixty seconds.

And the frontpage takes 15 seconds to load. That's 15 seconds of maxing out the CPU, causing lag on all other pages on the site, for every new visitor in the forum. Useless.

There must definitely be ways to optimize this - since you're not interested, I'm going to spend some time on it myself. First thing I'm going to do is look at the data requirements of the queries on the front page, and optimize the search index - I'm sure it must be positive to bring that down below 15 seconds ... basically, no page should take more than 1-2 seconds to load solo - that is, when they're the only page loading on the server. 1-2 seconds is already A LOT, even for a search page. And don't tell me it's impossible, just look at Google - it searches TWENTY MILLION pages in half a second, so of course there are ways to optimize this stuff. An optimized keyword search is of course only possible with actual words in the database though, so the first thing I'll do is change your script to use random words from an english dictionary...

By the way, the test forum you put up, do you use the experimental full-text index on the message text column? You might want to try that, to see how much of a difference it makes - it might already be enough, and if so, would certainly save me a lot of work ... only the indexes for the other pages would have to be optimized then ...

mindplay
just look at Google - it searches TWENTY MILLION pages in half a second
Sounds funny, because: a) Google is not using mySQL of course (and NOT ANY DATABASE at all, they are using their own data structure saved in a single files) b) Google have many hundreds, if not thousands, servers worldwide.

We for sure are interested in optimization, but as I said in very beginning - our forums is FOR SMALL AND MEDIUM sites, and it is exactly because of these reasons of optimization.

As about indexes and fulltext too - mySQL is not very optimized to work with them. I had, for example, one tourism database with just about 100,000 records - it was very well-structured and indexed, but these guys came to me and asked: what we did wrong? One simple select request costs us 22 seconds! The reason is if you combine statements like GROUP, ORDER, LIMIT or use altering tables like where Tb1.field1=Tb2.field2, mySQL will not as fine as it would be.

As I already said too - FULLTEXT might be a good idea, if it would be fully supported in mySQL. For now, it is only experimental stuff, and we have no idea what happens further. Developing a free program, we would not to make pain ourselves rewriting the script each time when database version changes (as it happens with PostgreSQL - fcuk, these guys are crazy, each new version has another SQL syntax).

You can try of course to optimize something with indexes in mySQL, and we would happy to know if it will help you (so we can use your excellent advices). But as I might expect, the solution is not to optimize mySQL. Solution is somewhere else.

MySQL can be used for this just as well as any database - it's already fast, you just need to optimize the way you use it ... yes, google use "their own data structures in a file" - a custom database. I don't need, nor do I expect to reach, Google-like performance - I want the forum to support maybe half a million posts without choking the server, that's all, I don't expect it to support twenty million records, or serve ten thousand users at a time.

But you're not going to get significantly better results by switching to SQLite, for example - even if it's twice as fast, that's nowhere NEAR fast enough.

But you're right, I'm not going to pull a solution out of my tall, black, magic hat. I'm sure there's lots of stuff that could be improved though - the indexing for one, if you didn't already optimize that? Caching of certain information, like the frontpage, which currently costs 15 seconds of CPU time for every new visitor. And of course the search, which is the real server-killer at the moment - perhaps a custom full-text index could be implemented and stored in cross-referencing tables in MySQL, possibly combined with caching too.

Whatever improvements I can make for LARGE sites, I'll try to keep it all scalable, so that performance/disk usage will not decrease on small/medium sites.

ANYTHING IS POSSIBLE, SO PLEASE QUIT TRYING TO DISCOURAGE ME! ;)

We are not discouraging you. Just do whatever you want.

mindplay
yepp, we're not trying to discourage :) if you will send me the script and sql changes - i'll try them, btw about server load you're not quite right this one hardly uses 10% of one of the cpus on this server, even when search runs on its full, cause server itself won't allow a php script to consume all resources - it just gives some, alloted by default to each script and it won't bring the server down, also mysql itself can't work faster .. it's a bottleneck even 4th version of it..


i'm just curious to see the results of your attempt.. just contact me - email me at my "sergei at gmx . net" email address and i'll make test of your modified search engine on 1 and half a million records.. that would be interesting!


thanks,

"this one hardly uses 10% of one of the cpus on this server" - what sort of weird load-balancing system are you using, that won't allow you to utilize all of your CPU power? .. I doubt it works that way, that wouldn't make any sense - it should utilize all of the CPU power, although of course is should distribute it evenly between running processes. Reducing the allowed CPU load to 10% while only one process is running wouldn't make ANY sense ;)

it's not weird :) the bottleneck in this case (heavy mysql load processing big database - it's 400 mb size), appears to be memory speed and hdd speed.

also note intel's hyperthreading which handles some multitasking stuff on one cpu - making 2 virtual cpus out of one.

bloody ages since i looked at it, but i remember using google and finding a list of useful heavy load stats for phpbb2, v-bulletin and a couple of other forums. if you've got five minutes it'd be worth trying to dig something like that out for comparison. if i've got time after work i'll have a look.

4days,

Comparisons do me little good at this time - I already chose MiniBB and completed the integration with our site, so I'm not really interested in how other boards perform ... more interesting is to make MiniBB perform better :) ... MiniBB does everything I need, most other boards are either too complex, too simple, or just bad ;) ... Besides, I wouldn't bet my hat on any of the larger, more complex boards handling large amounts of posts any better ...