Rotating Mysql Slow Query Log

Saturday, May 28, 2016

How To Rotating Mysql Slow Query Log :-


as assumptions.
1. Your mysql user is “mysql”
2. Your mysql slow query log is /var/log/mysql/mysql-slow.log

The script is written to perform the following actions:
1. Rotate weekly
2. Retain 3 rotations (3 files + live log)
3. Compress on rotate (gzip)
4. Create new logfile with 660 permissions chowned to mysql:mysql
5. Run: mysqladmin flush-logs

Note- Please be aware that the flush logs command will also rotate any binary logging currently in place, Note:- please ensure this will not adversely affect your deployment prior to use
Note:- Please ensure you carry out your own testing prior to deploying this script into a live environment.

#
# place this script in /etc/logrotate.d/ or your appropriate logrotate dir.
#
# NOTE: if you are reliant on binlogs i.e. for replication, 'flush logs' closes the current binlog and moves to the next itteration of log files
# You should test this does not cause an issue with your deployment before using this script

/var/log/mysql/mysql-slow.log {
    weekly
    rotate 3
    compress
    missingok
    notifempty
    sharedscripts
    create 660 mysql mysql
    postrotate
        /usr/bin/mysqladmin flush-logs
    endscript
}

Read more...

How can we audit a MySQL server ?

Saturday, May 14, 2016

How can we audit a MySQL server ?


Below are the various loging technologies available on a MySQL server for auditing :

1. The error log
2. The slow query log
3. The binary log
4. Custom made triggers
5. Using MySQL Proxy
6. The general  log


1. Using the error log :-

The error log contains information indicating when mysqld was started and stopped and also any critical errors.
The log_warnings system variable can be used to control warning logging to the error log.  If enabled aborted connections are written to the error log, and access-denied errors for new connection attempts are written.

mysql> set global log_warnings=2;
if we generate a connection with wrong password we get in the error log

$ mysql --protocol=TCP -hlocalhost -u root -P3309 -pxxx2

$ tail -f serge.err
...
130403 15:24:19 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Unfortunately the error log does not contain any information about the queries run against the database so it cannot be used as the basis of an auditing solution.

2. Using the he slow query log :-

Can we use the MySQL Slow  Query log as an audit solution ? It give some info but unfortunately this is not enough to be considered as an audit solution.
The slow query log consists of SQL statements that took more than long_query_time seconds to execute. The long _query_time can be set to 0 to log everything. Catching all queries through the slow query log can be costly as timing information is also collected and written.

mysql> set global slow_query_log=on;
mysql> set global long_query_time=0;
mysql> set global log_queries_not_using_indexes=on;
For a givent query we got in the log :

Time Id Command Argument
# Time: 140504 16:56:46
# User@Host: root[root] @ localhost [127.0.0.1] Id: 1
# Query_time: 0.492536 Lock_time: 0.032216 Rows_sent: 200 Rows_examined: 39222
use test;
SET timestamp=1466887444;
select * from (SELECT title, post_url,date(post_date) dt, author, MATCH(title,extract) AGAINST (' backup restore' in boolean mode) as Relevance FROM post where post_date > '2010-01-01'ORDER BY Relevance DESC limit 200) xx order by dt desc;
All statement are logged (select, insert update, delete). Connections are missing. We do not get  trace of failed requests.  There is no idea of filtering on users / schemas / tables. It is an everything or nothing mechanism. The impact on performance can be big.

3. Using binary log :-

The binary log contains only queries that have modified the data through commited transactions. So if a suspicious select is generated it will not appear here. This can however (if no other audit information is available) be used to discover when a modification was made. The binary log can be read an filtered to discover the modification.

4. Using triggers :-

Another sometimes used solution would be to use triggers on critical table. This solution is painful to maintain. Like with binary logs this only deals with data modification tracking. There is no trigger on connection or on select. Moreover as MySQL allow only one trigger per table this can become complicated for application requiring triggers for other purposes.

5. Using the MySQL Proxy :-

http://dev.mysql.com/doc/refman/5.6/en/mysql-proxy.html MySQL Proxy when it firs appeared has raised a lot of expectations. It can effectively be used to implement auditing, policy enforcement. Unfortunately the MySQL Proxy product has remained in the alpha status since many years. Because of this alpha status and total lack of visibility it cannot be part of a production long term solution.

6. Using the general log :-

The general log is a logging capability of the MySQL server allowing to log all queries recieved  by the server. As it logs all the queries it gives more info than the precedent techniques.

Let us try it

[user@user ~]$ mysql --protocol=TCP -hlocalhost -u root -P3306 -pmanager1
...
mysql> use test;
mysql> show tables;
mysql> create table titi(col1 int);
mysql> insert into titi values(3);
mysql> commit;
mysql> exit;

let do a failing connection :

[sfrezefo@serge ~]$ mysql --protocol=TCP -hlocalhost -u root -P3309 -pmanager2

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

This is what we get int the general log

Time Id Command Argument
130327 10:51:34 2 Connect root@localhost on
                2 Query select @@version_comment limit 1
130327 10:55:03 2 Quit
130327 10:55:07 3 Connect root@localhost on
                3 Query select @@version_comment limit 1
130327 10:55:15 3 Query SELECT DATABASE()
                3 Init DB test
                3 Query show databases
                3 Query show tables
                3 Field List planetpost
130327 10:55:26 3 Query show tables
130327 10:55:50 3 Query create table titi(col1 int)
130327 10:56:08 3 Query insert into titi values(3)
130327 10:56:13 3 Query commit
130327 13:33:41 3 Quit
130327 13:33:47 4 Connect root@localhost on

                4 Connect Access denied for user 'root'@'localhost' (using password: YES)

What is missing ? We  get the successful or failed connection. Subsequent queries are linked to the opened sessions  We get  trace of failed request except when syntactically invalid.  There is no idea of filtering on users/schemas/tables. It is an everything or nothing mechanism. The impact on performance can be big.

Conclusion :-


None of the technique presented carry enough information or flexibility to be considered as  auditing solutions.







Read more...

How to solve java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset?

How to solve java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset?

MariaDB server closing client connections unexpectedly ?

how to drop MySQL's autoReconnect=true from Confluence ?

Exceptions are blow --

java.sql.SQLNonTransientConnectionException: Could not read resultset: Connection reset
        at org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:136)
        at org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
        at org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:264)
        at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:288)
        at org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:302)
        at org.mariadb.jdbc.MySQLStatement.executeQuery(MySQLStatement.java:361)
        at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
        at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:208)
        at com.vimba.database.DBFactory.attemptLoginWithTempPasswordDetails(DBFactory.java:181)
        at com.vimba.database.DBFactory.authenticate(DBFactory.java:131)
        at com.vimba.service.ExposedFunctions.login(ExposedFunctions.java:88)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:210)
        at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:132)
        at com.sun.xml.ws.server.sei.EndpointMethodHandler.invoke(EndpointMethodHandler.java:241)
        at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:74)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:559)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:518)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:503)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:400)
        at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:226)
        at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:375)
        at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:175)
        at com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:134)
        at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:159)
        at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:49)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.mariadb.jdbc.internal.common.QueryException: Could not read resultset: Connection reset
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:926)
        at org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:991)
        at org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:281)
        ... 40 more
Caused by: java.net.SocketException: Connection reset


Solution is :-

ValidationQuery
Setting the validationQuery attribute to valid query for MySQL (e.g. "SELECT 1") prevents connection problems when using a data source in Tomcat (version tested is Tomcat 6.0.26 as used in Confluence 3.4 standalone) which is using DBCP.
autoReconnect is not required for this configuration.
                   
                        username="USER"
                        password="PASSWORD"
                        driverClassName="com.mysql.jdbc.Driver"
                        url="jdbc:mysql://localhost:3306/confluence?useUnicode=true&characterEncoding=utf8"
                        maxActive="15"
                        maxIdle="7"
                        validationQuery="Select 1" />

Note: Setting the validationQuery option on the database connection pool will have a performance impact as DBCP will use this query to validate connections before returning them to the caller.

Using autoReconnect without using the validationQuery does not prevent {{com.mysql.jdbc.exceptions.jdbc4.CommunicationsException}}s!


Or

you can use sessionVariables in the client's JDBC URL like so:
jdbc:mysql://hostname:3306/schema?sessionVariables=wait_timeout=600

u can try autoreconect also
jdbc:mysql://hostname:3306/schema?sessionVariables=wait_timeout=600&autoReconnect=true

Or

mysql://db_user:db_user@localhost/mydb?autoReconnect=true&useUnicode=yes



Read more...

SVN to GIT Migration Steps in BitBucket

Wednesday, May 11, 2016

SVN to GIT Migration Steps in BitBucket


Below are the Steps for migration from SVN to GIT


One time steps:-

Step 1. Install Git
            >apt-get install git

Step 2. Install GIT-SVN
           >sudo apt-get install git-svn

Step 3. Download svn migration JAR and place in root folder
            https://bitbucket.org/atlassian/svn-migration-scripts/downloads

Step 4. Check If all requirements are OK using verify command
            >java -jar ~/svn-migration-scripts.jar verify

This should give git version, svn version and git-svn version.



Steps to migrate a repository:-

Step 1. Export SVN authors
            java -jar ~/svn-migration-scripts.jar authors svn://192.16.32.10/repoName > authors.txt
or
            java -jar ~/svn-migration-scripts.jar authors svn://localhost/repoName > /usr/temp/repoNameAuthors.txt
This will create authors of SVN which will be mapped to GIT users during migration

Step 2. Change this authors file manually so that user names and email adresses are proper. Save this authors file. This file will be used while SVN to GIT conversion

Step 3. Convert SVN repository to GIT repository
git svn clone --stdlayout --authors-file= svn://
E.g.
git svn clone --stdlayout --authors-file=MyProjectauthors.txt svn://localhost/MyProject /root/GitProjects/MyProject

Step 4. Change directory to GIT repository
E.g. cd /root/GitProjects/MyProject

Step 5. Clean the tags. (To see which tags will be cleaned use below command. Note it will not make any changes)
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git

Step 6. Actually clean the tags.
java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

Step 7. Now a git repository is created which is on our local system. we need to import this on bitbucket

Step 8. Go to bitbucket and create a GIT repository

Step 9. On our system on command line change directory to git repository folder

Step 10. Add bitbucket repository as origin repository. From local system where we have converted repository execute this command
git remote add origin https://@bitbucket.org//.git
e.g.
git remote add origin https://gs51@bitbucket.org/gs51/MyProject.git

Step 10. Actually push the contents
              git push -u origin --all

Step 11. Push Tags
              git push --tags

Thanks

Read more...

About This Blog

Lorem Ipsum

  © Copyright 2009 Linux-HelpLine.Blogspot.com

Back to TOP