Høj load avarage på apache server
Jeg har en server, som kører noget apache.
Den får periodisk problemer med at load avarage stiger helt vildt. Nogen gange op over 80! Den har kun 1 CPU, så den burde jo gerne ligge under 1. hvad kan det dog være? Det hjælper at genstarte apache, men det er bare ikke nogen sund løsning at skulle genstarte på diverse random tidspunkter.
Serveren har 1 CPU på 2.2 GHZ og 4 GB ram.
jeg har vedhæftet et screenshot af nice top, dog har den ikke deet høje load avarage nu, men jeg synes ikke den ser meget anderledes ud når det sker, bortset fra at load avarage er mega høj, og serveren nærmest er umulig at komme i kontakt med.
Her er min apache2.conf - kan det være her der er noget galt?
Den får periodisk problemer med at load avarage stiger helt vildt. Nogen gange op over 80! Den har kun 1 CPU, så den burde jo gerne ligge under 1. hvad kan det dog være? Det hjælper at genstarte apache, men det er bare ikke nogen sund løsning at skulle genstarte på diverse random tidspunkter.
Serveren har 1 CPU på 2.2 GHZ og 4 GB ram.
jeg har vedhæftet et screenshot af nice top, dog har den ikke deet høje load avarage nu, men jeg synes ikke den ser meget anderledes ud når det sker, bortset fra at load avarage er mega høj, og serveren nærmest er umulig at komme i kontakt med.
Her er min apache2.conf - kan det være her der er noget galt?
#
# Based upon the NCSA server configuration files originally by Rob McCool.
#
# This is the main Apache server configuration file. It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.2/ for detailed information about
# the directives.
#
# Do NOT simply read the instructions in here without understanding
# what they do. They're here only as hints or reminders. If you are unsure
# consult the online docs. You have been warned.
#
# The configuration directives are grouped into three basic sections:
# 1. Directives that control the operation of the Apache server process as a
# whole (the 'global environment').
# 2. Directives that define the parameters of the 'main' or 'default' server,
# which responds to requests that aren't handled by a virtual host.
# These directives also provide default values for the settings
# of all virtual hosts.
# 3. Settings for virtual hosts, which allow Web requests to be sent to
# different IP addresses or hostnames and have them handled by the
# same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path. If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "foo.log"
# with ServerRoot set to "/etc/apache2" will be interpreted by the
# server as "/etc/apache2/foo.log".
#
### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#
#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE! If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at );
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"
#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
LockFile ${APACHE_LOCK_DIR}/accept.lock
#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300
#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On
#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100
#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15
##
## Server-Pool Size Regulation (MPM specific)
##
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a
# graceful restart. ThreadLimit can only be changed by stopping
# and starting Apache.
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxClients 150
MaxRequestsPerChild 0
# event MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestsPerChild 0
# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives. See also the AllowOverride
# directive.
#
AccessFileName .htaccess
#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
Order allow,deny
Deny from all
Satisfy all
#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value. If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain
#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a
# container, that host's errors will be logged there and not here.
#
ErrorLog ${APACHE_LOG_DIR}/error.log
#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn
# Include module configuration:
Include mods-enabled/*.load
Include mods-enabled/*.conf
# Include all the user configurations:
Include httpd.conf
# Include ports listing
Include ports.conf
#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
# If you are behind a reverse proxy, you might want to change %h into %{X-Forwarded-For}i
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.
# Include generic snippets of statements
Include conf.d/
# Include the virtual host configurations:
Include sites-enabled/
Kommentarer19
er du sikker på det er selve apache?
af en eller anden grund kan jeg ikke se top attached så tør ikke sige noget om hvilken process du skal kigge på.
Top er her, men den laver
top - 23:05:02 up 7:34, 2 users, load average: 0.41, 0.18, 0.05
Tasks: 89 total, 2 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.8%us, 1.0%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 4063552k total, 1251132k used, 2812420k free, 106180k buffers
Swap: 3781624k total, 0k used, 3781624k free, 563468k cached
Unknown command - try 'h' for help
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15891 www-data 20 0 225m 37m 4360 R 2.0 0.9 0:01.66 apache2
15890 www-data 20 0 225m 37m 4288 S 1.0 0.9 0:01.51 apache2
15794 www-data 20 0 219m 31m 4288 S 0.8 0.8 0:03.28 apache2
15844 www-data 20 0 225m 36m 5120 S 0.8 0.9 0:02.34 apache2
15888 www-data 20 0 225m 37m 4288 S 0.5 0.9 0:01.34 apache2
15893 www-data 20 0 225m 37m 4264 S 0.5 0.9 0:00.64 apache2
15931 root 20 0 154m 9.9m 6420 S 0.5 0.2 0:00.02 php
1041 root 20 0 63452 3152 2508 S 0.3 0.1 0:20.95 vmtoolsd
2451 root 30 10 19180 1452 1016 S 0.3 0.0 1:01.45 top
15719 www-data 20 0 225m 37m 4296 S 0.3 0.9 0:06.31 apache2
15809 www-data 20 0 225m 37m 4288 S 0.3 0.9 0:04.13 apache2
15842 www-data 20 0 219m 31m 4288 S 0.3 0.8 0:02.39 apache2
15889 www-data 20 0 219m 31m 3956 S 0.3 0.8 0:00.90 apache2
15928 root 30 10 19072 1332 1020 R 0.3 0.0 0:00.01 top
1 root 20 0 8356 816 684 S 0.0 0.0 0:01.95 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
Jeg har prøvet at hive det ud jeg kunne finde i apache errorlog.
PHP Fatal error: Call to a member function fetch_assoc() on a non-object in /var/www/ajax/np_index_stocks_xml.php on line 23, referer:
PHP Fatal error: Call to undefined function drupal_match_path() in /var/www/sites/all/modules/performance/performance.module on line 269
File does not exist: /var/www/favicon.ico
PHP Warning: mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on '10.0.0.13' (111) in /var/www/ajax/index_stocks_xml.php on line 5,
!!!Jeg tror den har lavet problemet omkring samme tid som de her fejl kommer!!!!
caught SIGTERM, shutting down
PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php5/20090626/php_mssql.dll' - /usr/lib/php5/20090626/php_mssql.dll: cannot open shared object file: No such file or directory in Unknown on line 0
Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze15 with Suhosin-Patch configured -- resuming normal operations
server reached MaxClients setting, consider raising the MaxClients setting
PHP Warning: mysqli::mysqli(): (HY000/2013): Lost connection to MySQL server at 'reading authorization packet', system error: 0 in /var/www/ajax/index_stocks_xml.php on line 5,
Mange af dem her
child process 18292 still did not exit, sending a SIGTERM
child process 18337 still did not exit, sending a SIGKILL
could not make child process 18094 exit, attempting to continue anyway
Giver det nogen mening?
2 ting der lige slår mig du
"server reached MaxClients setting, consider raising the MaxClients setting"
(mener denne setting ligger som vhost konfiguation.)
Derudover lader det til du har MySQL connection problems...
"
PHP Warning: mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on '10.0.0.13' (111) in /var/www/ajax/index_stocks_xml.php on line 5,
"
Jeg kan ikke se MySQL i processlisten, men den kører måske på en anden server?
Det ses nogengange, at hvis man ikke har DB forbindelser så ville connection poolen hobe sig op, og lave stor load for webserveren, da den aldrig får sluppet forbindelserne igen, før en evt. timeout trigger. (Kan være flere minutter)..
Jeg ville starte med MySQL delen. max-clients, kan være en følgeeffekt da den aldrig får sluppet forbindelserne opimod MySQL. Og tillade flere clients, vil umiddelbart ikke hjælpe på dit load, snare tværtimod.
Nåh ja, og så kan den høje load også være skyld i at din MySQL ikke svarer hvis den altså kører på denne server.. :)
Så det er en lidt trial and error..
caught SIGTERM,
caught SIGTERM, shutting down
Apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze15 with Suhosin-Patch configured -- resuming normal operations
server reached MaxClients setting, consider raising the MaxClients setting
PHP Warning: mysqli::mysqli(): (HY000/2013): Lost connection to MySQL server at 'reading authorization packet', system error: 0 in /var/www/ajax/index_stocks_xml.php on line 5,
Mange af dem her
child process 18292 still did not exit, sending a SIGTERM
child process 18337 still did not exit, sending a SIGKILL
could not make child process 18094 exit, attempting to continue anyway
Giver det nogen mening?
Ja tør ikke sige precis hvad der sker men noget kunne tyde på at du har en "side" der ikke lukker sine sessioner korrekt.
Så, nu laver den
I får lige en ny log oversigt og top. Det kunne godt tyde på at det er MySQL der giver problemet (denne er på en anden server, og den svarer - så det er lidt mystisk) jeg må kigge på den del først.
Log:
Mange af dem her: [Fri Apr 19 08:44:17 2013] [error] [client 194.239.212.211] PHP Warning: mysqli::mysqli(): (HY000/2003): Can't connect to MySQL server on '10.0.0.13' (110) in /var/www/ajax/np_index_stocks_xml.php on line 7
Efter noget tid, kommer den her, det stemmer også meget godt overens med hvad ibm siger:
[Fri Apr 19 08:43:41 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
Og dem her:
[Fri Apr 19 08:35:09 2013] [error] [client 87.48.86.254] PHP Fatal error: Call to undefined function drupal_match_path() in /var/www/sites/all/modules/performance/performance.module on line 269
top - 09:37:34 up 9:28, 1 user, load average: 42.40, 44.69, 37.17
Mem: 4063552k total, 4035380k used, 28172k free, 1296k buffers
Swap: 3781624k total, 822048k used, 2959576k free, 27492k cached
Unknown command - try 'h' for help
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12660 root 30 10 19192 1476 1020 R 9.2 0.0 0:00.11 top
12233 www-data 20 0 225m 29m 3796 S 4.6 0.8 0:01.21 apache2
12618 root 20 0 70588 3464 2660 S 4.6 0.1 0:00.11 sshd
1 root 20 0 8356 536 516 S 0.0 0.0 0:01.84 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root 20 0 0 0 0 S 0.0 0.0 0:00.81 events/0
7 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm
12 root 20 0 0 0 0 S 0.0 0.0 0:00.02 sync_supers
13 root 20 0 0 0 0 S 0.0 0.0 0:00.03 bdi-default
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0
15 root 20 0 0 0 0 S 0.0 0.0 0:00.51 kblockd/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kondemand/0
22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
23 root 20 0 0 0 0 D 0.0 0.0 0:03.67 kswapd0
24 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0
26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0
201 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0
202 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux
203 root 20 0 0 0 0 S 0.0 0.0 0:00.01 scsi_eh_0
204 root 20 0 0 0 0 S 0.0 0.0 0:00.02 scsi_eh_1
207 root 20 0 0 0 0 S 0.0 0.0 0:00.30 mpt_poll_0
208 root 20 0 0 0 0 S 0.0 0.0 0:00.00 mpt/0
210 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2
246 root 20 0 0 0 0 D 0.0 0.0 0:01.98 kjournald
290 root 20 0 0 0 0 D 0.0 0.0 0:00.37 flush-8:0
313 root 16 -4 16900 264 264 S 0.0 0.0 0:00.04 udevd
440 root 18 -2 16860 184 184 S 0.0 0.0 0:00.00 udevd
441 root 18 -2 16860 140 140 S 0.0 0.0 0:00.00 udevd
519 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
728 daemon 20 0 8096 324 324 S 0.0 0.0 0:00.00 portmap
749 statd 20 0 14384 520 520 S 0.0 0.0 0:00.00 rpc.statd
930 root 20 0 52556 996 784 S 0.0 0.0 0:00.47 rsyslogd
977 root 20 0 3920 376 376 S 0.0 0.0 0:00.00 acpid
1049 root 20 0 55256 1644 1352 S 0.0 0.0 0:19.64 vmtoolsd
1078 root 20 0 197m 4080 2448 S 0.0 0.1 0:03.84 apache2
1090 daemon 20 0 18716 240 240 S 0.0 0.0 0:00.00 atd
1116 root 20 0 22428 668 588 S 0.0 0.0 0:00.14 cron
1138 root 20 0 16672 184 184 S 0.0 0.0 0:00.00 daemon
1140 root 20 0 3956 452 452 S 0.0 0.0 0:00.07 mpt-statusd
1187 root 20 0 3956 380 380 S 0.0 0.0 0:00.00 mysqld_safe
1306 mysql 20 0 606m 45m 3072 D 0.0 1.2 2:08.28 mysqld
1307 root 20 0 3856 448 448 S 0.0 0.0 0:00.00 logger
1323 root 20 0 49176 588 476 S 0.0 0.0 0:00.01 sshd
Det passer tidsmæssigt med,
Kan en genstart af SQL serveren, få Apache til at gå helt i selvsving? Jeg synes ikke umiddelbart den har gjort det på samme tid før, men det kan ikke udelukkes for jeg har ikke været opmærksom på det.
Det kan være at MySQL
Som jeg ser det, kan serveren ikke forbinde til MySQL af den ene eller anden årsag.
Dette betyder at clients hober sig om, hvorefter at den smider fejlen med at du skal raise clients i Apache (hvilket egentlig ikke vil løse problemet).
Jeg ville kigge på hvorfor at webserveren ikke kan forbinde til MySQL serveren.
Hvilken log på MySQL
Disse 2 er helt tomme..
/var/log/mysql.err
/var/log/mysql.log
Jeg kan ærlig talt ikke
locate access.log
Hvis ikke SQL loggen er enabled:
http://serverfault.com/questions/71071/how-to-enable-mysql-logging
http://www.dbasquare.com/2012/06/02/how-to-find-mysql-binary-logs-error…
access.log ligger inde i
Det kan være at den ligger
Som sagt kan jeg ikke huske særlig meget konkret om MySQL. Det er der garanteret andre der kan.
En årsag til at mysql.log er tom, kunne meget vel være fordi den er disabled. Prøv om dette bringer dig videre:
http://stackoverflow.com/questions/6479107/how-to-enable-mysql-query-log
På debian ligger den
/var/log/mysql/mysql.log
Men skal slås til i /etc/mysql/my.cnf
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
Error loggen er her, og skal nok også slås til i samme fil.
log_error = /var/log/mysql/error.log
Min my.cnf har jeg rettet,
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
general_log_file = /var/log/mysql/mysql.log
general_log = 1
log_error = /var/log/mysql/error.log
log_error = 1
Jeg har oprettet de 2 filer, med 0666 rettigheder. Så burde den logge, right?
Den logger intet, skal man genstarte MySQL før det fungerer?
Den logger intet, skal
Ohh yes. Du skal altid genstarte en service efter konfiguration. Genstart maskinen eller (så vidt jeg husker):
sudo service mysql start
Jeg stoppede den general log
Errorlog kommer der intet i, heller ikke selvom jeg bevidst kører en syntax der fejler. Så den er nok ikke aktiveret korrekt.
Ved faktisk ikke helt 100%
Anyhow, 20.000 operationer i sekundet. Har du tjekket processlisten og antal connections i MySQL'en når du løber ind i disse problemer?
Jeg har lavet en nice top,
Kan man se hvor meget hver af dens CPU'er er presset?
I kan lige få en top, den kører masser af operationer lige nu.
top - 10:23:45 up 54 min, 2 users, load average: 0.49, 0.77, 0.73
Tasks: 89 total, 2 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.2%us, 0.9%sy, 0.0%ni, 84.3%id, 3.8%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 4063280k total, 1552820k used, 2510460k free, 437624k buffers
Swap: 904184k total, 0k used, 904184k free, 432208k cached
Hej,
Hvis du trykker 1 når
Hvis du trykker 1 når top er aktiv skulle du kunne se de individuelle cpu'er.
Ok, I får den lige her med
top - 12:05:02 up 2:35, 2 users, load average: 1.64, 1.72, 1.51
Tasks: 89 total, 1 running, 88 sleeping, 0 stopped, 0 zombie
Cpu0 : 12.2%us, 3.3%sy, 0.0%ni, 58.1%id, 26.1%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu1 : 5.9%us, 0.3%sy, 0.0%ni, 76.1%id, 17.6%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 6.6%us, 0.0%sy, 0.0%ni, 60.4%id, 33.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4063280k total, 1644580k used, 2418700k free, 464204k buffers
Swap: 904184k total, 0k used, 904184k free, 462452k cached